You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (Created) (JIRA)" <ji...@apache.org> on 2012/02/07 23:37:00 UTC
[jira] [Created] (HBASE-5349) Automagically tweak global memstore
and block cache sizes based on workload
Automagically tweak global memstore and block cache sizes based on workload
---------------------------------------------------------------------------
Key: HBASE-5349
URL: https://issues.apache.org/jira/browse/HBASE-5349
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Fix For: 0.94.0
Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/
That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore
and block cache sizes based on workload
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215228#comment-13215228 ]
Jean-Daniel Cryans commented on HBASE-5349:
-------------------------------------------
And by waiting to reach the lower barrier with a constant heavy load, you'll always run into HBASE-5161.
> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
> Key: HBASE-5349
> URL: https://issues.apache.org/jira/browse/HBASE-5349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0
>
>
> Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore
and block cache sizes based on workload
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257103#comment-13257103 ]
Zhihong Yu commented on HBASE-5349:
-----------------------------------
This is a plan worth pursuing.
> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
> Key: HBASE-5349
> URL: https://issues.apache.org/jira/browse/HBASE-5349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0
>
>
> Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore
and block cache sizes based on workload
Posted by "Enis Soztutar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257085#comment-13257085 ]
Enis Soztutar commented on HBASE-5349:
--------------------------------------
I have been thinking about this, and I think we can have a shot at a simple implementation. Let me summarize what I have in mind before starting the implementation:
Goals:
- Provide min - max heap percentages for block cache (memstore kind of has it). I think we should keep max-min sanity bounds, and if they are equal, disable auto-tuning.
- enable optimizing the available memory for adaptive workloads (mostly writes during the day, a lot of reads once MR job starts, etc). For example, when a large write job is started after ~10 minutes, region servers should tune for write workload.
Non-goals:
- find the optimum mem-utilization algorithm
- introduce a bunch of other parameters, to get rid of the current ones
- make it very experimental so that nobody enables it in production.
Ideally, to optimize the usage of the available memory, we should predict the future workload (possibly from past workload), and devise a model capturing all the costs associated with block cache hits / misses, flushes, compactions, etc. But this model will be very complex to do it properly.
I have checked Hypertable's implementation, and it seems that they check whether the load is read/write heavy by some hard coded values for the counters, and increment/decrement the mem limits, much like what Zhihong proposes above. I also want to start with something similar.
Implementation layer:
- Currently global memstore limit is a soft limit, we may have to make it a hard limit (blocking writes)
- we should enable incrementing / decrementing and setting global memstore and block cache maximum limits. We do not have live configuration changes, but regardless of auto-tuning, we should be able to manually set those online.
- Periodically we should check past workload (like past 10 min), and depending on whether it is write heavy or read heavy (from metrics), adjust the mem limits in small intervals.
What do you guys think? Still worth pursuing?
> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
> Key: HBASE-5349
> URL: https://issues.apache.org/jira/browse/HBASE-5349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0
>
>
> Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore
and block cache sizes based on workload
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257219#comment-13257219 ]
stack commented on HBASE-5349:
------------------------------
I wouldn't mind more detail.
Can our LRU be resized?
Memstore upper bound can vary but there are interesting effects like if its too big, flushing can take so long, the memstore fills before we get around to flushing it again so we block.
Nit: 10 minutes seems like too coarse a granularity?
Good stuff Enis.
> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
> Key: HBASE-5349
> URL: https://issues.apache.org/jira/browse/HBASE-5349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0
>
>
> Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore
and block cache sizes based on workload
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204087#comment-13204087 ]
Jean-Daniel Cryans commented on HBASE-5349:
-------------------------------------------
Good question, I don't think looking at requests is good enough... instead we could look at how both are used and if there's adjustment to be made. For example, if you have a read heavy workload then the memstores would not see a lot of usage... same with write heavy, the block cache would be close to empty.
Those two are clear cuts, now for those workloads in between it gets a bit harder. Maybe at first we shouldn't even try to optimize them.
I think it should also be done incrementally, move like 3-5% of the heap from one place to the other every few minutes until it settles.
> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
> Key: HBASE-5349
> URL: https://issues.apache.org/jira/browse/HBASE-5349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0
>
>
> Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore
and block cache sizes based on workload
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204325#comment-13204325 ]
Zhihong Yu commented on HBASE-5349:
-----------------------------------
I am thinking of introducing rolling counters (using circular buffer) for both MemStore and LruBlockCache.
We record the number of flushes (for MemStore) and evictions (for LruBlockCache), respectively.
Every 1 (configurable) minute, we roll the counters.
It should be straight forward to observe whether MemStore or LruBlockCache is under pressure by looking at the trend of the rolling counters.
Once we determine the one under pressure, we can utilize what J-D described above to shift heap among the two.
We can also introduce weights between MemStore and LruBlockCache for the rolling counters.
> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
> Key: HBASE-5349
> URL: https://issues.apache.org/jira/browse/HBASE-5349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0
>
>
> Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore
and block cache sizes based on workload
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204078#comment-13204078 ]
Zhihong Yu commented on HBASE-5349:
-----------------------------------
w.r.t. the measure for determining workload, should the measure be computed solely based on one region server ?
Or should this measure be relative to the workload on other region servers ?
> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
> Key: HBASE-5349
> URL: https://issues.apache.org/jira/browse/HBASE-5349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0
>
>
> Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore
and block cache sizes based on workload
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204070#comment-13204070 ]
Zhihong Yu commented on HBASE-5349:
-----------------------------------
We currently don't maintain moving average of read/write requests per region server.
What should be an effective measure for determining read vs. write heavy workload ?
> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
> Key: HBASE-5349
> URL: https://issues.apache.org/jira/browse/HBASE-5349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0
>
>
> Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5349) Automagically tweak global memstore
and block cache sizes based on workload
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5349:
---------------------------------
Fix Version/s: (was: 0.94.0)
0.96.0
Moving out of 0.94.
> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
> Key: HBASE-5349
> URL: https://issues.apache.org/jira/browse/HBASE-5349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0
>
>
> Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5349) Automagically tweak global memstore
and block cache sizes based on workload
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5349:
-------------------------
Fix Version/s: (was: 0.96.0)
Moving out of 0.96.0
> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
> Key: HBASE-5349
> URL: https://issues.apache.org/jira/browse/HBASE-5349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
>
> Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore
and block cache sizes based on workload
Posted by "Mubarak Seyed (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204740#comment-13204740 ]
Mubarak Seyed commented on HBASE-5349:
--------------------------------------
For reference:
Cassandra 1.0.0 supports self-tunes memtable sizes ([CASSANDRA-2787|https://issues.apache.org/jira/browse/CASSANDRA-2787]). It uses Java agent ([Jamm|https://github.com/jbellis/jamm])
> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
> Key: HBASE-5349
> URL: https://issues.apache.org/jira/browse/HBASE-5349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0
>
>
> Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore
and block cache sizes based on workload
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214944#comment-13214944 ]
Jean-Daniel Cryans commented on HBASE-5349:
-------------------------------------------
@Mubarak
We already have that through HeapSize, it's really just a matter of knowing what to auto-tune and when.
> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
> Key: HBASE-5349
> URL: https://issues.apache.org/jira/browse/HBASE-5349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0
>
>
> Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore
and block cache sizes based on workload
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215220#comment-13215220 ]
stack commented on HBASE-5349:
------------------------------
Chatting w/ J-D about a phenomenon where we do not use memory when we are taking on a bunch of writes w/ a low region count.
The few regions we have grow to their max of 128M or so and then we flush but in his case he had gigs of free memory still. The notion is that we should let memstores grow to fill all available space and then flush when they hit the low-water global mem mark for the memstore.
The problem then becomes we'll flush lots of massive files and will overwhelm compactions. We'll need a push-back, something like a flush-merge where we flush by rewriting an existing store file interleaving the contents of memory or some such to slow down the flush but also to make for less compaction to do.
> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
> Key: HBASE-5349
> URL: https://issues.apache.org/jira/browse/HBASE-5349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0
>
>
> Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira