You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "LN (JIRA)" <ji...@apache.org> on 2008/07/15 11:43:31 UTC
[jira] Updated: (HBASE-745) scaling of one regionserver, improving
memory and cpu usage
[ https://issues.apache.org/jira/browse/HBASE-745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
LN updated HBASE-745:
---------------------
Description:
after weeks testing hbase 0.1.3 and hadoop(0.16.4, 0.17.1), i found there are many works to do, before a particular regionserver can handle data about 100G, or even more. i'd share my opions here with stack, and other developers.
first, the easiest way improving scalability of regionserver is upgrading hardware, use 64bit os and 8G memory for the regionserver process, and speed up disk io.
besides hardware, following are software bottlenecks i found in regionserver:
1. as data increasing, compaction was eating cpu(with io) times, the total compaction time is basicly linear relative to whole data size, even worse, sometimes square relavtive to that size.
2. memory usage are depends on opened mapfiles
3. network connection are depends on opened mapfiles, see HADOOP-2341 and HBASE-24.
was:
after weeks testing hbase 0.1.3 and hadoop(0.16.4, 0.17.1), i found there are many works to do, before a particular regionserver can handle data about 100G, or even more. i'd share my opions here with stack, and other developers.
first, the easiest way improving scalability of regionserver is upgrading hardware, use 64bit os and 8G memory for the regionserver process, and speed up disk io.
besides hardware, following are software bottlenecks i found in regionserver:
1. as data increasing, compaction was eating cpu(with io) times, the total compaction time is basicly linear relative to whole data size, even worse, sometimes square relavtive to that size.
2. memory and socket connection usage are depends on opened mapfiles, see HADOOP-2341 and HBASE-24.
will explain above in comments later.
> scaling of one regionserver, improving memory and cpu usage
> -----------------------------------------------------------
>
> Key: HBASE-745
> URL: https://issues.apache.org/jira/browse/HBASE-745
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.1.3
> Environment: hadoop 0.17.1
> Reporter: LN
> Priority: Minor
>
> after weeks testing hbase 0.1.3 and hadoop(0.16.4, 0.17.1), i found there are many works to do, before a particular regionserver can handle data about 100G, or even more. i'd share my opions here with stack, and other developers.
> first, the easiest way improving scalability of regionserver is upgrading hardware, use 64bit os and 8G memory for the regionserver process, and speed up disk io.
> besides hardware, following are software bottlenecks i found in regionserver:
> 1. as data increasing, compaction was eating cpu(with io) times, the total compaction time is basicly linear relative to whole data size, even worse, sometimes square relavtive to that size.
> 2. memory usage are depends on opened mapfiles
> 3. network connection are depends on opened mapfiles, see HADOOP-2341 and HBASE-24.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.