You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Mikhail Bautin (Created) (JIRA)" <ji...@apache.org> on 2012/03/19 22:07:39 UTC
[jira] [Created] (HBASE-5601) Add per-column-family data block
cache hit ratios
Add per-column-family data block cache hit ratios
-------------------------------------------------
Key: HBASE-5601
URL: https://issues.apache.org/jira/browse/HBASE-5601
Project: HBase
Issue Type: Improvement
Reporter: Mikhail Bautin
In addition to the overall block cache hit ratio it would be extremely useful to have per-column-family data block cache hit ratio metrics.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5601) Add per-column-family data block
cache hit ratios
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249147#comment-13249147 ]
jiraposter@reviews.apache.org commented on HBASE-5601:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4675/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
HBASE-5601. Add per-column-family data block cache hit ratios. First shot at the patch.
Changes are as follows (some are backwards incompatible):
- Renamed hbase.RegionServerDynamicStatistics.XXX -> hbase.regionserver.dyn.XXX
- Renamed the metrics name generation from
hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt
-> tbl=usertable.cf=ycsb.bt=Data.at=scan.blockCacheHitCount
Notice that tbl.TableName is changed to tbl=TableName, cf.Family is
changed to cf=Family, the fs or compaction prefix is changed to
at=scan or at=compaction. at stands for access type.
- Through using CacheAccessStats/CacheSizeStats, metric names
across per-cf and global are made consistent.
- For compaction, per-block type metrics are disabled. For compaction
we do not need to distinguish between block types.
- Added a configuration "hbase.metrics.dynamic.enabled" for disabling
dynamic metrics. It is enabled by default.
- Changed bloom filter metrics from keyMaybeInBloomCnt -> bloomHitCount
- Removed some code for keeping BC related to META block types
Some changes in HFileReaders are due to automatic trailing whitespace
settings in my eclipse. I can regenerate those parts if it is hard to
review.
This addresses bug HBASE-5601.
https://issues.apache.org/jira/browse/HBASE-5601
Diffs
-----
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java e3c4fe4
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessMetrics.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStatsImpl.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2ba3ab2
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeMetrics.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStatsImpl.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java ef03075
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStatsImpl.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java e952ef9
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 79ffee0
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java aa99add
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 91562f7
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 87634ba
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 1795229
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 251ccd4
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java f1a68e0
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java 3b46dd8
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java da08532
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java 5ad7406
src/main/resources/hbase-default.xml 44ee689
src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java f7430ee
src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java 5f8214e
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java d980948
src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java 17284a3
src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java bd83a71
src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java dbe425f
Diff: https://reviews.apache.org/r/4675/diff
Testing
-------
Tested with existing unit tests, and manually using FileContext
Thanks,
enis
> Add per-column-family data block cache hit ratios
> -------------------------------------------------
>
> Key: HBASE-5601
> URL: https://issues.apache.org/jira/browse/HBASE-5601
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Enis Soztutar
>
> In addition to the overall block cache hit ratio it would be extremely useful to have per-column-family data block cache hit ratio metrics.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5601) Add per-column-family data block
cache hit ratios
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249148#comment-13249148 ]
jiraposter@reviews.apache.org commented on HBASE-5601:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4675/
-----------------------------------------------------------
(Updated 2012-04-07 03:40:41.839514)
Review request for hbase.
Changes
-------
Fixed a failing test case.
Summary
-------
HBASE-5601. Add per-column-family data block cache hit ratios. First shot at the patch.
Changes are as follows (some are backwards incompatible):
- Renamed hbase.RegionServerDynamicStatistics.XXX -> hbase.regionserver.dyn.XXX
- Renamed the metrics name generation from
hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt
-> tbl=usertable.cf=ycsb.bt=Data.at=scan.blockCacheHitCount
Notice that tbl.TableName is changed to tbl=TableName, cf.Family is
changed to cf=Family, the fs or compaction prefix is changed to
at=scan or at=compaction. at stands for access type.
- Through using CacheAccessStats/CacheSizeStats, metric names
across per-cf and global are made consistent.
- For compaction, per-block type metrics are disabled. For compaction
we do not need to distinguish between block types.
- Added a configuration "hbase.metrics.dynamic.enabled" for disabling
dynamic metrics. It is enabled by default.
- Changed bloom filter metrics from keyMaybeInBloomCnt -> bloomHitCount
- Removed some code for keeping BC related to META block types
Some changes in HFileReaders are due to automatic trailing whitespace
settings in my eclipse. I can regenerate those parts if it is hard to
review.
This addresses bug HBASE-5601.
https://issues.apache.org/jira/browse/HBASE-5601
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2ba3ab2
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java e3c4fe4
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessMetrics.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStatsImpl.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeMetrics.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStatsImpl.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java ef03075
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStatsImpl.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java e952ef9
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 79ffee0
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java aa99add
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 91562f7
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 87634ba
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 1795229
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 251ccd4
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java f1a68e0
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java 3b46dd8
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java da08532
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java 5ad7406
src/main/resources/hbase-default.xml 44ee689
src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java f7430ee
src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java 5f8214e
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java d980948
src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java 17284a3
src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java bd83a71
src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java dbe425f
Diff: https://reviews.apache.org/r/4675/diff
Testing
-------
Tested with existing unit tests, and manually using FileContext
Thanks,
enis
> Add per-column-family data block cache hit ratios
> -------------------------------------------------
>
> Key: HBASE-5601
> URL: https://issues.apache.org/jira/browse/HBASE-5601
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Enis Soztutar
>
> In addition to the overall block cache hit ratio it would be extremely useful to have per-column-family data block cache hit ratio metrics.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5601) Add per-column-family data block
cache hit ratios
Posted by "Enis Soztutar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244822#comment-13244822 ]
Enis Soztutar commented on HBASE-5601:
--------------------------------------
I have started working on this (and other per-table-cf metrics), but soon realized that some of the changes require breaking compatibility with (soon to be released) 0.94 in terms of metric names. So I decided to list the proposed changes first to see whether we can come up with a plan.
As you know, HBASE-4219 and others added per-cf metrics, but there are a couple of things we can improve on that (great) work. Some ideas I have:
- Make Per-cf and global block cache metrics naming consistent (hbase.regionserver.blockCacheHitCount vs hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt)
- Rename hbase.RegionServerDynamicStatistics.XXX -> hbase.regionserver.dyn.XXX
- Rename dynamic metric names tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt -> tbl=usertable.cf=ycsb.bt=Data.at=scan.BlockCacheHitCount. Here at stands for access type, and can be scan or compaction.
- For at=compaction, we probably do not need per-block type (Data, Index, ...) metric values, shall we aggregate them?
- Shall we extend per-cf metrics into HBASE-5533 (read/write latency histograms)?
- With slab cache, we may no longer have one global block cache. Shall we enable per-cache type metrics? This can imply renaming hbase.regionserver.blockCache => hbase.regionserver.lrublockcache, slabblockcache, etc.
- Add a configuration for disabling per-cf metrics.
- Thoughts ?
> Add per-column-family data block cache hit ratios
> -------------------------------------------------
>
> Key: HBASE-5601
> URL: https://issues.apache.org/jira/browse/HBASE-5601
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
>
> In addition to the overall block cache hit ratio it would be extremely useful to have per-column-family data block cache hit ratio metrics.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5601) Add per-column-family data block
cache hit ratios
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Otis Gospodnetic updated HBASE-5601:
------------------------------------
Component/s: monitoring
metrics
> Add per-column-family data block cache hit ratios
> -------------------------------------------------
>
> Key: HBASE-5601
> URL: https://issues.apache.org/jira/browse/HBASE-5601
> Project: HBase
> Issue Type: Improvement
> Components: metrics, monitoring
> Reporter: Mikhail Bautin
> Assignee: Enis Soztutar
>
> In addition to the overall block cache hit ratio it would be extremely useful to have per-column-family data block cache hit ratio metrics.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5601) Add per-column-family data block
cache hit ratios
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245469#comment-13245469 ]
stack commented on HBASE-5601:
------------------------------
bq. Make Per-cf and global block cache metrics naming consistent (hbase.regionserver.blockCacheHitCount vs hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt)
Is it really named the latter? Please fix (Proposal looks good).
bq. Rename hbase.RegionServerDynamicStatistics.XXX -> hbase.regionserver.dyn.XXX
hbase-5701 just committed did this (sort-of).
bq. Rename dynamic metric names tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt -> tbl=usertable.cf=ycsb.bt=Data.at=scan.BlockCacheHitCount. Here at stands for access type, and can be scan or compaction.
I don't follow the above. Something didn't make it across.
bq. For at=compaction, we probably do not need per-block type (Data, Index, ...) metric values, shall we aggregate them?
Sure, on compaction.
bq. Shall we extend per-cf metrics into HBASE-5533 (read/write latency histograms)?
That would be sweet but maybe for later.
bq. With slab cache, we may no longer have one global block cache. Shall we enable per-cache type metrics? This can imply renaming hbase.regionserver.blockCache => hbase.regionserver.lrublockcache, slabblockcache, etc.
Having them distinctly named makes sense to me.
bq. Add a configuration for disabling per-cf metrics.
Yes. There could be an overwhelming amount.
ba. Thoughts?
Sounds excellent. Sounds like work that would be checked in with the singularity, 0.96?
> Add per-column-family data block cache hit ratios
> -------------------------------------------------
>
> Key: HBASE-5601
> URL: https://issues.apache.org/jira/browse/HBASE-5601
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Enis Soztutar
>
> In addition to the overall block cache hit ratio it would be extremely useful to have per-column-family data block cache hit ratio metrics.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5601) Add per-column-family data block
cache hit ratios
Posted by "Enis Soztutar (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Enis Soztutar reassigned HBASE-5601:
------------------------------------
Assignee: Enis Soztutar
> Add per-column-family data block cache hit ratios
> -------------------------------------------------
>
> Key: HBASE-5601
> URL: https://issues.apache.org/jira/browse/HBASE-5601
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Enis Soztutar
>
> In addition to the overall block cache hit ratio it would be extremely useful to have per-column-family data block cache hit ratio metrics.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira