You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/01/16 17:16:34 UTC
[jira] Created: (HADOOP-2624) [hbase] memory management
[hbase] memory management
-------------------------
Key: HADOOP-2624
URL: https://issues.apache.org/jira/browse/HADOOP-2624
Project: Hadoop
Issue Type: Improvement
Components: contrib/hbase
Reporter: stack
Each Store has a Memcache of edits that is flushed on a fixed period (It used to be flushed when it grew beyond a limit). A Region can be made up of N Stores. A regionserver has no upper bound on the number of regions that can be deployed to it currently. Add to this that per mapfile, we have read the index into memory. We're also talking about adding caching of blocks and cells.
We need a means of keeping an account of memory usage adjusting cache sizes and flush rates (or sizes) dynamically -- using References where possible -- to accomodate deployment of added regions. If memory is strained, we should reject regions proffered by the master with a resouce-constrained, or some such, message.
The manual sizing we currently do ain't going to cut it for clusters of any decent size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2624) [hbase] memory management
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman reassigned HADOOP-2624:
-------------------------------------
Assignee: (was: Jim Kellerman)
> [hbase] memory management
> -------------------------
>
> Key: HADOOP-2624
> URL: https://issues.apache.org/jira/browse/HADOOP-2624
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
>
> Each Store has a Memcache of edits that is flushed on a fixed period (It used to be flushed when it grew beyond a limit). A Region can be made up of N Stores. A regionserver has no upper bound on the number of regions that can be deployed to it currently. Add to this that per mapfile, we have read the index into memory. We're also talking about adding caching of blocks and cells.
> We need a means of keeping an account of memory usage adjusting cache sizes and flush rates (or sizes) dynamically -- using References where possible -- to accomodate deployment of added regions. If memory is strained, we should reject regions proffered by the master with a resouce-constrained, or some such, message.
> The manual sizing we currently do ain't going to cut it for clusters of any decent size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2624) [hbase] memory management
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560035#action_12560035 ]
Jim Kellerman commented on HADOOP-2624:
---------------------------------------
Memory usage should be factored into HServerLoad, then the master can make more intelligent decisions about where to deploy regions.
> [hbase] memory management
> -------------------------
>
> Key: HADOOP-2624
> URL: https://issues.apache.org/jira/browse/HADOOP-2624
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
>
> Each Store has a Memcache of edits that is flushed on a fixed period (It used to be flushed when it grew beyond a limit). A Region can be made up of N Stores. A regionserver has no upper bound on the number of regions that can be deployed to it currently. Add to this that per mapfile, we have read the index into memory. We're also talking about adding caching of blocks and cells.
> We need a means of keeping an account of memory usage adjusting cache sizes and flush rates (or sizes) dynamically -- using References where possible -- to accomodate deployment of added regions. If memory is strained, we should reject regions proffered by the master with a resouce-constrained, or some such, message.
> The manual sizing we currently do ain't going to cut it for clusters of any decent size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2624) [hbase] memory management
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman reassigned HADOOP-2624:
-------------------------------------
Assignee: Jim Kellerman
> [hbase] memory management
> -------------------------
>
> Key: HADOOP-2624
> URL: https://issues.apache.org/jira/browse/HADOOP-2624
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Jim Kellerman
>
> Each Store has a Memcache of edits that is flushed on a fixed period (It used to be flushed when it grew beyond a limit). A Region can be made up of N Stores. A regionserver has no upper bound on the number of regions that can be deployed to it currently. Add to this that per mapfile, we have read the index into memory. We're also talking about adding caching of blocks and cells.
> We need a means of keeping an account of memory usage adjusting cache sizes and flush rates (or sizes) dynamically -- using References where possible -- to accomodate deployment of added regions. If memory is strained, we should reject regions proffered by the master with a resouce-constrained, or some such, message.
> The manual sizing we currently do ain't going to cut it for clusters of any decent size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2624) [hbase] memory management
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560089#action_12560089 ]
Billy Pearson commented on HADOOP-2624:
---------------------------------------
+1
This will also help with clusters that servers with varying memory sizes in region servers.
> [hbase] memory management
> -------------------------
>
> Key: HADOOP-2624
> URL: https://issues.apache.org/jira/browse/HADOOP-2624
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
>
> Each Store has a Memcache of edits that is flushed on a fixed period (It used to be flushed when it grew beyond a limit). A Region can be made up of N Stores. A regionserver has no upper bound on the number of regions that can be deployed to it currently. Add to this that per mapfile, we have read the index into memory. We're also talking about adding caching of blocks and cells.
> We need a means of keeping an account of memory usage adjusting cache sizes and flush rates (or sizes) dynamically -- using References where possible -- to accomodate deployment of added regions. If memory is strained, we should reject regions proffered by the master with a resouce-constrained, or some such, message.
> The manual sizing we currently do ain't going to cut it for clusters of any decent size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.