You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2011/09/12 23:07:09 UTC
[jira] [Created] (HBASE-4374) Up default regions size from 256M to
1G
Up default regions size from 256M to 1G
---------------------------------------
Key: HBASE-4374
URL: https://issues.apache.org/jira/browse/HBASE-4374
Project: HBase
Issue Type: Task
Reporter: stack
Priority: Blocker
Fix For: 0.92.0
HBASE-4365 has some discussion of why we default for a table should tend to fewer bigger regions. It doesn't look like this issue will be done for 0.92. For 0.92, lets up default region size from 256M to 1G and talk up pre-split on table creation in manual.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4374) Up default regions size from 256M
to 1G
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135007#comment-13135007 ]
Hudson commented on HBASE-4374:
-------------------------------
Integrated in HBase-TRUNK #2366 (See [https://builds.apache.org/job/HBase-TRUNK/2366/])
HBASE-4374 Up default regions size from 256M to 1G
stack :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/resources/hbase-default.xml
> Up default regions size from 256M to 1G
> ---------------------------------------
>
> Key: HBASE-4374
> URL: https://issues.apache.org/jira/browse/HBASE-4374
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4374.txt
>
>
> HBASE-4365 has some discussion of why we default for a table should tend to fewer bigger regions. It doesn't look like this issue will be done for 0.92. For 0.92, lets up default region size from 256M to 1G and talk up pre-split on table creation in manual.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4374) Up default regions size from 256M
to 1G
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134622#comment-13134622 ]
Jean-Daniel Cryans commented on HBASE-4374:
-------------------------------------------
+1
> Up default regions size from 256M to 1G
> ---------------------------------------
>
> Key: HBASE-4374
> URL: https://issues.apache.org/jira/browse/HBASE-4374
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Priority: Blocker
> Fix For: 0.92.0
>
>
> HBASE-4365 has some discussion of why we default for a table should tend to fewer bigger regions. It doesn't look like this issue will be done for 0.92. For 0.92, lets up default region size from 256M to 1G and talk up pre-split on table creation in manual.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4374) Up default regions size from 256M
to 1G
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106935#comment-13106935 ]
stack commented on HBASE-4374:
------------------------------
So, I think going to 1G, we first need to test a little. We might want to up the default flush size while we are at it. In old days there used to be an optimal balance to be struck between flush size, compaction size and region split size. Let me do some testing before submitting a patch here.
> Up default regions size from 256M to 1G
> ---------------------------------------
>
> Key: HBASE-4374
> URL: https://issues.apache.org/jira/browse/HBASE-4374
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Priority: Blocker
> Fix For: 0.92.0
>
>
> HBASE-4365 has some discussion of why we default for a table should tend to fewer bigger regions. It doesn't look like this issue will be done for 0.92. For 0.92, lets up default region size from 256M to 1G and talk up pre-split on table creation in manual.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4374) Up default regions size from 256M
to 1G
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103397#comment-13103397 ]
Andrew Purtell commented on HBASE-4374:
---------------------------------------
If we can get online schema edits into 0.92, at least metadata like HTD/HCD attributes, then users can easily change the split points as tables grow. Can add a ruby script helper for this purpose in bin or shell support. Does not take away from talking up presplitting but provides a good alternative if presplitting is not an option for whatever reason (e.g. keyspace distribution not well known).
> Up default regions size from 256M to 1G
> ---------------------------------------
>
> Key: HBASE-4374
> URL: https://issues.apache.org/jira/browse/HBASE-4374
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Priority: Blocker
> Fix For: 0.92.0
>
>
> HBASE-4365 has some discussion of why we default for a table should tend to fewer bigger regions. It doesn't look like this issue will be done for 0.92. For 0.92, lets up default region size from 256M to 1G and talk up pre-split on table creation in manual.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4374) Up default regions size from 256M to
1G
Posted by "stack (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-4374.
--------------------------
Resolution: Fixed
Release Note: Upped default split size to 1G from 256M and the flush size from 64M to 128M.
> Up default regions size from 256M to 1G
> ---------------------------------------
>
> Key: HBASE-4374
> URL: https://issues.apache.org/jira/browse/HBASE-4374
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4374.txt
>
>
> HBASE-4365 has some discussion of why we default for a table should tend to fewer bigger regions. It doesn't look like this issue will be done for 0.92. For 0.92, lets up default region size from 256M to 1G and talk up pre-split on table creation in manual.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4374) Up default regions size from 256M
to 1G
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134729#comment-13134729 ]
stack commented on HBASE-4374:
------------------------------
Committed to branch and trunk.
> Up default regions size from 256M to 1G
> ---------------------------------------
>
> Key: HBASE-4374
> URL: https://issues.apache.org/jira/browse/HBASE-4374
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4374.txt
>
>
> HBASE-4365 has some discussion of why we default for a table should tend to fewer bigger regions. It doesn't look like this issue will be done for 0.92. For 0.92, lets up default region size from 256M to 1G and talk up pre-split on table creation in manual.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-4374) Up default regions size
from 256M to 1G
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103397#comment-13103397 ]
Andrew Purtell edited comment on HBASE-4374 at 9/13/11 6:25 AM:
----------------------------------------------------------------
If we can get online schema edits into 0.92, at least metadata like HTD/HCD attributes, then users can easily change the region size split threshold as tables grow. Can add a ruby script helper for this purpose in bin or shell support. Does not take away from talking up presplitting but provides a good alternative if presplitting is not an option for whatever reason (e.g. keyspace distribution not well known).
was (Author: apurtell):
If we can get online schema edits into 0.92, at least metadata like HTD/HCD attributes, then users can easily change the split points as tables grow. Can add a ruby script helper for this purpose in bin or shell support. Does not take away from talking up presplitting but provides a good alternative if presplitting is not an option for whatever reason (e.g. keyspace distribution not well known).
> Up default regions size from 256M to 1G
> ---------------------------------------
>
> Key: HBASE-4374
> URL: https://issues.apache.org/jira/browse/HBASE-4374
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Priority: Blocker
> Fix For: 0.92.0
>
>
> HBASE-4365 has some discussion of why we default for a table should tend to fewer bigger regions. It doesn't look like this issue will be done for 0.92. For 0.92, lets up default region size from 256M to 1G and talk up pre-split on table creation in manual.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4374) Up default regions size from 256M
to 1G
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135325#comment-13135325 ]
Jean-Daniel Cryans commented on HBASE-4374:
-------------------------------------------
Shouldn't the defaults in the code be changed too? Stuff like this in HConstants:
{code}
HConstants:
/** Default maximum file size */
public static final long DEFAULT_MAX_FILE_SIZE = 256 * 1024 * 1024;
HTableDescriptor:
public static final long DEFAULT_MEMSTORE_FLUSH_SIZE = 1024*1024*64L;
{code}
> Up default regions size from 256M to 1G
> ---------------------------------------
>
> Key: HBASE-4374
> URL: https://issues.apache.org/jira/browse/HBASE-4374
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4374.txt
>
>
> HBASE-4365 has some discussion of why we default for a table should tend to fewer bigger regions. It doesn't look like this issue will be done for 0.92. For 0.92, lets up default region size from 256M to 1G and talk up pre-split on table creation in manual.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4374) Up default regions size from 256M
to 1G
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134618#comment-13134618 ]
stack commented on HBASE-4374:
------------------------------
So, i'm going to make a patch that does 1G regions and 128M flushes (with the multiplier still of x2 so we'll grow to 256M before the barrier comes down). Any objections? (A 2GB memstore flush is fine when lzo is on but we want something for default).
> Up default regions size from 256M to 1G
> ---------------------------------------
>
> Key: HBASE-4374
> URL: https://issues.apache.org/jira/browse/HBASE-4374
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Priority: Blocker
> Fix For: 0.92.0
>
>
> HBASE-4365 has some discussion of why we default for a table should tend to fewer bigger regions. It doesn't look like this issue will be done for 0.92. For 0.92, lets up default region size from 256M to 1G and talk up pre-split on table creation in manual.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4374) Up default regions size from 256M to
1G
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4374:
-------------------------
Attachment: 4374.txt
Flush size of 128M and split size of 1G.
> Up default regions size from 256M to 1G
> ---------------------------------------
>
> Key: HBASE-4374
> URL: https://issues.apache.org/jira/browse/HBASE-4374
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4374.txt
>
>
> HBASE-4365 has some discussion of why we default for a table should tend to fewer bigger regions. It doesn't look like this issue will be done for 0.92. For 0.92, lets up default region size from 256M to 1G and talk up pre-split on table creation in manual.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4374) Up default regions size from 256M
to 1G
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135114#comment-13135114 ]
Hudson commented on HBASE-4374:
-------------------------------
Integrated in HBase-0.92 #79 (See [https://builds.apache.org/job/HBase-0.92/79/])
HBASE-4374 Up default regions size from 256M to 1G
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/resources/hbase-default.xml
> Up default regions size from 256M to 1G
> ---------------------------------------
>
> Key: HBASE-4374
> URL: https://issues.apache.org/jira/browse/HBASE-4374
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4374.txt
>
>
> HBASE-4365 has some discussion of why we default for a table should tend to fewer bigger regions. It doesn't look like this issue will be done for 0.92. For 0.92, lets up default region size from 256M to 1G and talk up pre-split on table creation in manual.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4374) Up default regions size from 256M
to 1G
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127829#comment-13127829 ]
Jean-Daniel Cryans commented on HBASE-4374:
-------------------------------------------
bq. In old days there used to be an optimal balance to be struck between flush size, compaction size and region split size.
IMO something that radically screws this up is compression. I've been experimenting with 2GB memstore flush size on a LZO'd table and saw radically better write performance.
I think it would be good that we bring the flush size up as we bring up the split size, since this means less regions per RS leaving room for bigger and more efficient Memstores.
> Up default regions size from 256M to 1G
> ---------------------------------------
>
> Key: HBASE-4374
> URL: https://issues.apache.org/jira/browse/HBASE-4374
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Priority: Blocker
> Fix For: 0.92.0
>
>
> HBASE-4365 has some discussion of why we default for a table should tend to fewer bigger regions. It doesn't look like this issue will be done for 0.92. For 0.92, lets up default region size from 256M to 1G and talk up pre-split on table creation in manual.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira