You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org> on 2012/10/11 22:51:06 UTC

[jira] [Created] (HBASE-6981) multiple commit logs per region server

Kannan Muthukkaruppan created HBASE-6981:
--------------------------------------------

             Summary: multiple commit logs per region server
                 Key: HBASE-6981
                 URL: https://issues.apache.org/jira/browse/HBASE-6981
             Project: HBase
          Issue Type: New Feature
            Reporter: Kannan Muthukkaruppan
            Assignee: Kannan Muthukkaruppan


For write dominated workloads, the fact that we have only 1 commit log per server  artificially limits the aggregate throughput to that of one disk (this is especially true if we do true FS level syncs on each datanode, or even frequent intermittent FS level syncs as data is being written to each block).

We could consider allowing a configurable number of commit logs per server (perhaps something close to or slightly less than the number of disks per server), and we could shard regions on the server, by maybe just a simple modulo scheme, into those commit logs.

[A quick way to experiment with the same might be to run multiple region server instances per server; but might be better operationally to package the feature into a single region server.]



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6981) multiple commit logs per region server

Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kannan Muthukkaruppan updated HBASE-6981:
-----------------------------------------

    Assignee: Liyin Tang  (was: Kannan Muthukkaruppan)
    
> multiple commit logs per region server
> --------------------------------------
>
>                 Key: HBASE-6981
>                 URL: https://issues.apache.org/jira/browse/HBASE-6981
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Liyin Tang
>
> For write dominated workloads, the fact that we have only 1 commit log per server  artificially limits the aggregate throughput to that of one disk (this is especially true if we do true FS level syncs on each datanode, or even frequent intermittent FS level syncs as data is being written to each block).
> We could consider allowing a configurable number of commit logs per server (perhaps something close to or slightly less than the number of disks per server), and we could shard regions on the server, by maybe just a simple modulo scheme, into those commit logs.
> [A quick way to experiment with the same might be to run multiple region server instances per server; but might be better operationally to package the feature into a single region server.]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6981) multiple commit logs per region server

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474500#comment-13474500 ] 

Ted Yu commented on HBASE-6981:
-------------------------------

Seems to have the same idea from HBASE-5699
                
> multiple commit logs per region server
> --------------------------------------
>
>                 Key: HBASE-6981
>                 URL: https://issues.apache.org/jira/browse/HBASE-6981
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Kannan Muthukkaruppan
>
> For write dominated workloads, the fact that we have only 1 commit log per server  artificially limits the aggregate throughput to that of one disk (this is especially true if we do true FS level syncs on each datanode, or even frequent intermittent FS level syncs as data is being written to each block).
> We could consider allowing a configurable number of commit logs per server (perhaps something close to or slightly less than the number of disks per server), and we could shard regions on the server, by maybe just a simple modulo scheme, into those commit logs.
> [A quick way to experiment with the same might be to run multiple region server instances per server; but might be better operationally to package the feature into a single region server.]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6981) multiple commit logs per region server

Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475264#comment-13475264 ] 

Kannan Muthukkaruppan commented on HBASE-6981:
----------------------------------------------

Ted: You are right. It is pretty much the same. Liyin is planning to take a look at this issue.
                
> multiple commit logs per region server
> --------------------------------------
>
>                 Key: HBASE-6981
>                 URL: https://issues.apache.org/jira/browse/HBASE-6981
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Kannan Muthukkaruppan
>
> For write dominated workloads, the fact that we have only 1 commit log per server  artificially limits the aggregate throughput to that of one disk (this is especially true if we do true FS level syncs on each datanode, or even frequent intermittent FS level syncs as data is being written to each block).
> We could consider allowing a configurable number of commit logs per server (perhaps something close to or slightly less than the number of disks per server), and we could shard regions on the server, by maybe just a simple modulo scheme, into those commit logs.
> [A quick way to experiment with the same might be to run multiple region server instances per server; but might be better operationally to package the feature into a single region server.]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-6981) multiple commit logs per region server

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack resolved HBASE-6981.
--------------------------

    Resolution: Duplicate

Resolving as dup of HBASE-5699.  There is more action over on that issue.
                
> multiple commit logs per region server
> --------------------------------------
>
>                 Key: HBASE-6981
>                 URL: https://issues.apache.org/jira/browse/HBASE-6981
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Liyin Tang
>
> For write dominated workloads, the fact that we have only 1 commit log per server  artificially limits the aggregate throughput to that of one disk (this is especially true if we do true FS level syncs on each datanode, or even frequent intermittent FS level syncs as data is being written to each block).
> We could consider allowing a configurable number of commit logs per server (perhaps something close to or slightly less than the number of disks per server), and we could shard regions on the server, by maybe just a simple modulo scheme, into those commit logs.
> [A quick way to experiment with the same might be to run multiple region server instances per server; but might be better operationally to package the feature into a single region server.]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira