You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org> on 2012/10/11 22:51:06 UTC
[jira] [Created] (HBASE-6981) multiple commit logs per region
server
Kannan Muthukkaruppan created HBASE-6981:
--------------------------------------------
Summary: multiple commit logs per region server
Key: HBASE-6981
URL: https://issues.apache.org/jira/browse/HBASE-6981
Project: HBase
Issue Type: New Feature
Reporter: Kannan Muthukkaruppan
Assignee: Kannan Muthukkaruppan
For write dominated workloads, the fact that we have only 1 commit log per server artificially limits the aggregate throughput to that of one disk (this is especially true if we do true FS level syncs on each datanode, or even frequent intermittent FS level syncs as data is being written to each block).
We could consider allowing a configurable number of commit logs per server (perhaps something close to or slightly less than the number of disks per server), and we could shard regions on the server, by maybe just a simple modulo scheme, into those commit logs.
[A quick way to experiment with the same might be to run multiple region server instances per server; but might be better operationally to package the feature into a single region server.]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6981) multiple commit logs per region
server
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kannan Muthukkaruppan updated HBASE-6981:
-----------------------------------------
Assignee: Liyin Tang (was: Kannan Muthukkaruppan)
> multiple commit logs per region server
> --------------------------------------
>
> Key: HBASE-6981
> URL: https://issues.apache.org/jira/browse/HBASE-6981
> Project: HBase
> Issue Type: New Feature
> Reporter: Kannan Muthukkaruppan
> Assignee: Liyin Tang
>
> For write dominated workloads, the fact that we have only 1 commit log per server artificially limits the aggregate throughput to that of one disk (this is especially true if we do true FS level syncs on each datanode, or even frequent intermittent FS level syncs as data is being written to each block).
> We could consider allowing a configurable number of commit logs per server (perhaps something close to or slightly less than the number of disks per server), and we could shard regions on the server, by maybe just a simple modulo scheme, into those commit logs.
> [A quick way to experiment with the same might be to run multiple region server instances per server; but might be better operationally to package the feature into a single region server.]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6981) multiple commit logs per region
server
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474500#comment-13474500 ]
Ted Yu commented on HBASE-6981:
-------------------------------
Seems to have the same idea from HBASE-5699
> multiple commit logs per region server
> --------------------------------------
>
> Key: HBASE-6981
> URL: https://issues.apache.org/jira/browse/HBASE-6981
> Project: HBase
> Issue Type: New Feature
> Reporter: Kannan Muthukkaruppan
> Assignee: Kannan Muthukkaruppan
>
> For write dominated workloads, the fact that we have only 1 commit log per server artificially limits the aggregate throughput to that of one disk (this is especially true if we do true FS level syncs on each datanode, or even frequent intermittent FS level syncs as data is being written to each block).
> We could consider allowing a configurable number of commit logs per server (perhaps something close to or slightly less than the number of disks per server), and we could shard regions on the server, by maybe just a simple modulo scheme, into those commit logs.
> [A quick way to experiment with the same might be to run multiple region server instances per server; but might be better operationally to package the feature into a single region server.]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6981) multiple commit logs per region
server
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475264#comment-13475264 ]
Kannan Muthukkaruppan commented on HBASE-6981:
----------------------------------------------
Ted: You are right. It is pretty much the same. Liyin is planning to take a look at this issue.
> multiple commit logs per region server
> --------------------------------------
>
> Key: HBASE-6981
> URL: https://issues.apache.org/jira/browse/HBASE-6981
> Project: HBase
> Issue Type: New Feature
> Reporter: Kannan Muthukkaruppan
> Assignee: Kannan Muthukkaruppan
>
> For write dominated workloads, the fact that we have only 1 commit log per server artificially limits the aggregate throughput to that of one disk (this is especially true if we do true FS level syncs on each datanode, or even frequent intermittent FS level syncs as data is being written to each block).
> We could consider allowing a configurable number of commit logs per server (perhaps something close to or slightly less than the number of disks per server), and we could shard regions on the server, by maybe just a simple modulo scheme, into those commit logs.
> [A quick way to experiment with the same might be to run multiple region server instances per server; but might be better operationally to package the feature into a single region server.]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6981) multiple commit logs per region
server
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-6981.
--------------------------
Resolution: Duplicate
Resolving as dup of HBASE-5699. There is more action over on that issue.
> multiple commit logs per region server
> --------------------------------------
>
> Key: HBASE-6981
> URL: https://issues.apache.org/jira/browse/HBASE-6981
> Project: HBase
> Issue Type: New Feature
> Reporter: Kannan Muthukkaruppan
> Assignee: Liyin Tang
>
> For write dominated workloads, the fact that we have only 1 commit log per server artificially limits the aggregate throughput to that of one disk (this is especially true if we do true FS level syncs on each datanode, or even frequent intermittent FS level syncs as data is being written to each block).
> We could consider allowing a configurable number of commit logs per server (perhaps something close to or slightly less than the number of disks per server), and we could shard regions on the server, by maybe just a simple modulo scheme, into those commit logs.
> [A quick way to experiment with the same might be to run multiple region server instances per server; but might be better operationally to package the feature into a single region server.]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira