You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Doug Meil (JIRA)" <ji...@apache.org> on 2011/07/12 22:48:59 UTC
[jira] [Created] (HBASE-4089) UI for blockCache contents report
UI for blockCache contents report
---------------------------------
Key: HBASE-4089
URL: https://issues.apache.org/jira/browse/HBASE-4089
Project: HBase
Issue Type: New Feature
Reporter: Doug Meil
A UI that would display a block-cache report for a RegionServer would be helpful. For example ...
table1
cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
table2
cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
... Etc.
The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
No such interface exists in HRegionInterface, so this is not just a UI request also an API change. But I think it would be helpful from an operational perspective.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072809#comment-13072809 ]
Doug Meil commented on HBASE-4089:
----------------------------------
Regarding dumping the summary report to the log, I think exposing a public 'printSummary' (logSummary?) method on LruBlockCache would do it. Another thread can take care of the scheduling on how often the block cache summary should be run.
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084377#comment-13084377 ]
Jean-Daniel Cryans commented on HBASE-4089:
-------------------------------------------
Move this out of the method:
{code}
final String pattern = "\\" + HFile.CACHE_KEY_SEPARATOR;
{code}
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf, java_blockcache_checkpoint_2011_08_11.patch
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444421#comment-13444421 ]
Doug Meil commented on HBASE-4089:
----------------------------------
Unless somebody already did the UI showing the details of the block cache report (by table, etc.) then it's not done.
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf, java_blockcache_checkpoint_2011_08_11.patch
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Meil reassigned HBASE-4089:
--------------------------------
Assignee: Doug Meil
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Ming Ma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083671#comment-13083671 ]
Ming Ma commented on HBASE-4089:
--------------------------------
useful doc, Doug.
It seems like this one, https://issues.apache.org/jira/browse/HBASE-4147, https://issues.apache.org/jira/browse/HBASE-4145 need some common infrastructure to log and analyze structured data.
1. RS Web UI is useful. But that only provides the most recent value.
2. As you mentioned in the doc, we can create a static metric for each combination of table and CF. That could end up with lots of metrics. Might not be ideal.
3. How we plan to analyze the data is an important factor for the design.
a. Is there a latency requirement? In a production system, it is better to get these reports sooner than later.
b. Is it easy to do query and analysis on the data?, e.g., aggregate, max, etc.
4. Some ideas along the line of custom output
a. Can the log data be asynchronously uploaded to a special table in hbase? It might be a bit strange to upload data back to hbase. However, for performance, we can partition the special table into regions so that each region is colocated on the same RS where the log is generated; no automatic compaction, split, load balancing on the table.
b. Upload the log to HDFS periodically. Run map reduce jobs to mine the data with a customized inputformat. This might be ok if there is no strong latency requirement.
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf, java_blockcache_checkpoint_2011_08_11.patch
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082640#comment-13082640 ]
stack commented on HBASE-4089:
------------------------------
I'd say the name just has to be guaranteed unique... short would be nice too.
Currently name is made in here for both v1 and v2 hfiles:
{code}
public static String getBlockCacheKey(String hfileName, long offset) {
return hfileName + CACHE_KEY_SEPARATOR + offset;
}
{code}
The hfilename seems to depend on the fact storefile names are unique across hbase. They are made here using the 'name' part of the full storefile Path (from the AbstractHFileReader constructor):
this.name = path.getName();
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444241#comment-13444241 ]
Doug Meil commented on HBASE-4089:
----------------------------------
The sub-task HBASE-4200 was closed as a dup, what was the ticket that implemented that UI?
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf, java_blockcache_checkpoint_2011_08_11.patch
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Meil updated HBASE-4089:
-----------------------------
Description:
Summarized block-cache report for a RegionServer would be helpful. For example ...
table1
cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
table2
cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
... Etc.
The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
was:
A UI that would display a block-cache report for a RegionServer would be helpful. For example ...
table1
cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
table2
cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
... Etc.
The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
No such interface exists in HRegionInterface, so this is not just a UI request also an API change. But I think it would be helpful from an operational perspective.
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084290#comment-13084290 ]
Jean-Daniel Cryans commented on HBASE-4089:
-------------------------------------------
Some comments on the patch:
- Writables need a default constructor! See my 01/Aug/11 18:03 comment ;)
- About the style, I'd prefer you don't put extra white spaces like in this snippet:
{code}
arg0.writeUTF( table );
{code}
- Also pay attention to those lines > 80 chars.
- Shouldn't the pattern in getBlockCacheSummary be a member of the class instead? (and still final)
- Try to set the size of containers when you already know it, like at the end of getBlockCacheSummary
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf, java_blockcache_checkpoint_2011_08_11.patch
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084347#comment-13084347 ]
Doug Meil commented on HBASE-4089:
----------------------------------
Thanks! I'll make the changes.
re: "default constructor"
Yep, you said that already. Doh! Although in my doh-fense I hadn't gotten to the RS-API yet. :-)
re: "Shouldn't the pattern in getBlockCacheSummary be a member of the class instead? (and still final)"
I'm not sure I know what you mean...
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf, java_blockcache_checkpoint_2011_08_11.patch
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083689#comment-13083689 ]
Doug Meil commented on HBASE-4089:
----------------------------------
Yep, I logged HBASE-4147 and did that writeup too. :-) Summarizing and periodically outputting data is a theme. I think that writing these summaries back into an hbase table is a good idea. I personally don't think this kind of data fits well into what are currently called 'metrics' which is much higher level.
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf, java_blockcache_checkpoint_2011_08_11.patch
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083145#comment-13083145 ]
Doug Meil commented on HBASE-4089:
----------------------------------
Ahhh.. ok, I get it. This... "8351478435190657655_0" ... makes sense now.
Since the first part is a StoreFile, I can use directory paths to figure out which CF, which region, and which table this StoreFile belongs to because the directory structure is /table/region/cf/storefile. I'll basically construct a Map where the key is the hfileName (aka StoreFile) and the value is an object that contains table/cf (since that is the level that the report needs to roll up to).
I'm currently not aware of any utility that has this kind of lookup, but I think I now understand how to build it.
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073638#comment-13073638 ]
Doug Meil commented on HBASE-4089:
----------------------------------
Thanks JD. I included the JMX/Ganglia/et al. because it came up as suggestions in the dist-list, but I really didn't see how it would fit with this type of usage reporting. I'm glad you came to the same conclusion!
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13076004#comment-13076004 ]
Andrew Purtell commented on HBASE-4089:
---------------------------------------
bq. Does that mean that BlockCacheSummary returned from BlockCache should implement writable, or is there another class that represents the BlockCacheSummary that implements Writable that has the same information
Objects sent over RPC implement Writable directly, so by that convention BlockCacheSummary should implement Writable.
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444243#comment-13444243 ]
Doug Meil commented on HBASE-4089:
----------------------------------
Realistically, I don't think I can do the front-end work.
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf, java_blockcache_checkpoint_2011_08_11.patch
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Meil updated HBASE-4089:
-----------------------------
Attachment: hbase_4089_blockcachereport.pdf
Adding writeup of use-cases and 1st-pass general design.
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Meil updated HBASE-4089:
-----------------------------
Attachment: java_blockcache_checkpoint_2011_08_11.patch
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf, java_blockcache_checkpoint_2011_08_11.patch
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081815#comment-13081815 ]
stack commented on HBASE-4089:
------------------------------
Tell us more about this reverse lookup.... whats that going to look like? Are you sure what you are seeing is not a filename plus an offset?
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079523#comment-13079523 ]
Jean-Daniel Cryans commented on HBASE-4089:
-------------------------------------------
+1
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082387#comment-13082387 ]
Doug Meil commented on HBASE-4089:
----------------------------------
Let me back up a bit on this... what is the contract on the 'name' attribute in CachedBlock (the value class of the internal block cache map)? What is that supposed to be?
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079349#comment-13079349 ]
Doug Meil commented on HBASE-4089:
----------------------------------
Ok, how about this for the next course of action:
Subtask #1: implement basic summary in BlockCache/LruBlockCache, RS-API change, web-UI.
Subtask #2: this is the "output to X" part of the ticket.
I'll start with #1, hold off on #2 for now.
Does that sound reasonable?
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443538#comment-13443538 ]
Lars Hofhansl commented on HBASE-4089:
--------------------------------------
Doug, are you planning on finishing this?
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf, java_blockcache_checkpoint_2011_08_11.patch
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084381#comment-13084381 ]
Doug Meil commented on HBASE-4089:
----------------------------------
Sure thing. I didn't want it in the loop obviously, but it didn't seem like it was important enough to be permanently in memory in the block cache (although it's pretty small).
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf, java_blockcache_checkpoint_2011_08_11.patch
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072812#comment-13072812 ]
Doug Meil commented on HBASE-4089:
----------------------------------
If this approach is acceptable, probably should add this to the BlockCache interface. This is how the block cache is accessed.
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080509#comment-13080509 ]
Doug Meil commented on HBASE-4089:
----------------------------------
Hmmm.. started implementation and have been running other unit tests and getting more information on blockNames in the cache, and here's what they look like now: "8351478435190657655_0". That looks a lot more like a "block" than what I had documented in the writeup, but unfortunately I have nothing else to go on. Based on what I see, I need to do a reverse lookup in the catalog for the containing StoreFile (to hopefully get full-path, where I can get table/CF).
Anybody know any easy way to do that? :-)
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Meil updated HBASE-4089:
-----------------------------
Description:
A UI that would display a block-cache report for a RegionServer would be helpful. For example ...
table1
cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
table2
cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
... Etc.
The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
No such interface exists in HRegionInterface, so this is not just a UI request also an API change. But I think it would be helpful from an operational perspective.
was:
A UI that would display a block-cache report for a RegionServer would be helpful. For example ...
table1
cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
table2
cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
... Etc.
The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
No such interface exists in HRegionInterface, so this is not just a UI request also an API change. But I think it would be helpful from an operational perspective.
Summary: blockCache contents report (was: UI for blockCache contents report)
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
>
> A UI that would display a block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface, so this is not just a UI request also an API change. But I think it would be helpful from an operational perspective.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073635#comment-13073635 ]
Jean-Daniel Cryans commented on HBASE-4089:
-------------------------------------------
Nice document Doug, it puts everyone else to shame :)
I don't think we can expose those metrics through JMX/Ganglia/OpenTSDB as they will be changing a lot. It would be "doable" only if the regions and families never changed IMO. I'd prefer we concentrate on presenting this information from inside HBase.
In the nice to haves I'd like to see:
- Number of accesses/misses per block or family (could see what's hot, well cached, etc)
- Total size of the family on disk (then you can tell what portion of the dataset you cached)
Regarding the Writable question, you have to do that because it's required by Hadoop RPC. Since you are adding new infos, you'll have to implement it. Don't forget the default constructor! :)
For the web UI, what about making the region name clickable?
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444285#comment-13444285 ]
Lars Hofhansl commented on HBASE-4089:
--------------------------------------
Maybe I closed the sub task in error. Looking at the Web UI, I can see block cache statistics. But this is different. Should I reopen?
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf, java_blockcache_checkpoint_2011_08_11.patch
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083670#comment-13083670 ]
Doug Meil commented on HBASE-4089:
----------------------------------
Hey JD/Stack, can you guys sniff this patch? This is only a checkpoint - not the final product - but it actually works (I have a hacked up unit test that isn't in this patch). What I hope to be the hard-part hopefully (i.e., the summarization) is done, now it needs to be added to the RS API.
> blockCache contents report
> --------------------------
>
> Key: HBASE-4089
> URL: https://issues.apache.org/jira/browse/HBASE-4089
> Project: HBase
> Issue Type: New Feature
> Reporter: Doug Meil
> Assignee: Doug Meil
> Attachments: hbase_4089_blockcachereport.pdf, java_blockcache_checkpoint_2011_08_11.patch
>
>
> Summarized block-cache report for a RegionServer would be helpful. For example ...
> table1
> cf1 100 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 200 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> table2
> cf1 75 blocks, totalBytes=yyyyy, averageTimeInCache=XXXX hours
> cf2 150 blocks, totalBytes=zzzzz, averageTimeInCache=XXXX hours
> ... Etc.
> The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important.
> No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective.
> Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira