You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "mingjian (JIRA)" <ji...@apache.org> on 2011/06/29 14:41:28 UTC
[jira] [Created] (HBASE-4044) [performance]decrease IO calls when
flush hlog
[performance]decrease IO calls when flush hlog
----------------------------------------------
Key: HBASE-4044
URL: https://issues.apache.org/jira/browse/HBASE-4044
Project: HBase
Issue Type: Improvement
Components: performance, regionserver
Reporter: mingjian
Fix For: 0.90.4
Attachments: HLog.patch
We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to
disk once it gets "updateLock".
In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "mingjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
mingjian updated HBASE-4044:
----------------------------
Attachment: (was: HLog.patch)
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Assignee: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog-trunk-v1.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to
> disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "mingjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059316#comment-13059316 ]
mingjian commented on HBASE-4044:
---------------------------------
yes...because hlog is sequencefile and use append to write. We can't append more than 1 record now.
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Assignee: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog-trunk-v1.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu resolved HBASE-4044.
---------------------------
Resolution: Won't Fix
Assignee: (was: mingjian)
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog-trunk-v1.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "mingjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
mingjian updated HBASE-4044:
----------------------------
Attachment: HLog.patch
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to
> disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "mingjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
mingjian updated HBASE-4044:
----------------------------
Attachment: HLog-trunk-v1.patch
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Assignee: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog-trunk-v1.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to
> disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057343#comment-13057343 ]
Ted Yu commented on HBASE-4044:
-------------------------------
I ran test suite which got stuck in TestSplitTransactionOnCluster
I will wait for other developers' comment(s) since today is Hadoop Summit.
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Assignee: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog-trunk-v1.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "Jean-Daniel Cryans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-4044:
--------------------------------------
Fix Version/s: (was: 0.90.4)
Removing the fix version as it wasn't fixed anywhere.
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Attachments: HLog-trunk-v1.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "mingjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060423#comment-13060423 ]
mingjian commented on HBASE-4044:
---------------------------------
yep
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Assignee: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog-trunk-v1.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "mingjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057598#comment-13057598 ]
mingjian commented on HBASE-4044:
---------------------------------
Oh...It's not really decrease IO calls.
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Assignee: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog-trunk-v1.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu reassigned HBASE-4044:
-----------------------------
Assignee: mingjian
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Assignee: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to
> disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057446#comment-13057446 ]
Andrew Purtell commented on HBASE-4044:
---------------------------------------
This sounds reasonable.
I'll try out the patch.
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Assignee: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog-trunk-v1.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "mingjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
mingjian updated HBASE-4044:
----------------------------
Description:
We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to disk once it gets "updateLock".
In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
was:
We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to
disk once it gets "updateLock".
In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Assignee: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog-trunk-v1.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059654#comment-13059654 ]
Ted Yu commented on HBASE-4044:
-------------------------------
@mingjian:
Shall I close this JIRA ?
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Assignee: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog-trunk-v1.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058876#comment-13058876 ]
stack commented on HBASE-4044:
------------------------------
What do you mean Mingjian? The patch makes no difference?
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Assignee: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog-trunk-v1.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057230#comment-13057230 ]
Ted Yu commented on HBASE-4044:
-------------------------------
Great findings mingjian.
Do we need the check here ?
{code}
+ private void changeToFlush(){
+ if(flushEdits.isEmpty() == false)
+ flushEdits.clear();
{code}
We can call ArrayList.addAll() later in the routine.
I prefer clearing flushEdits after its contents are flushed for better GC. This is my personal opinion.
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to
> disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4044) [performance]decrease IO calls when
flush hlog
Posted by "mingjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057246#comment-13057246 ]
mingjian commented on HBASE-4044:
---------------------------------
Thanks for Ted. I modified the codes and generated the patch from trunk.
> [performance]decrease IO calls when flush hlog
> ----------------------------------------------
>
> Key: HBASE-4044
> URL: https://issues.apache.org/jira/browse/HBASE-4044
> Project: HBase
> Issue Type: Improvement
> Components: performance, regionserver
> Reporter: mingjian
> Assignee: mingjian
> Fix For: 0.90.4
>
> Attachments: HLog-trunk-v1.patch
>
>
> We found that the hlog flush to disk would block other write threads. When one thread exec "doWrite(info, logKey, edit);", the others wait for "updateLock" in HLog.java.
> Why not the others add their edits into a list and wait. When sync's time, the whole list sync to disk once. I think it will decrease the IO calls.
> So Maybe we will make two lists for edits. Each thread write to the "waledits" and wait for "updateLock". Each thread can copy the "waledits" to "flushedits" and flush the "flushedits" to
> disk once it gets "updateLock".
> In my test, it can increase the write speed of 100% when I set "hbase.regionserver.handler.count"=100.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira