You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jaehui-Lee (via GitHub)" <gi...@apache.org> on 2025/11/26 04:06:10 UTC
[PR] HBASE-29731 Add L1 and L2 cache hit/miss caching metrics [hbase]
Jaehui-Lee opened a new pull request, #7482:
URL: https://github.com/apache/hbase/pull/7482
https://issues.apache.org/jira/browse/HBASE-29731
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29731 Add L1 and L2 cache hit/miss caching metrics [hbase]
Posted by "Jaehui-Lee (via GitHub)" <gi...@apache.org>.
Jaehui-Lee commented on PR #7482:
URL: https://github.com/apache/hbase/pull/7482#issuecomment-3581654722
@wchevreuil
Thank you for the review! I apologize for the insufficient explanation in my initial description. Let me provide additional clarification.
> And can you explain further what you mean by "non-cached reads (such as those from compactions)" ? I don't think compactions bypass the cache, AFAIK, it use same HFileReaderImpl in it's scanner, which always looks the cache first.
You're absolutely right! Compactions do read from the cache first. What I meant to convey is that while compaction reads do check the cache first, they generate many cache misses, which are reflected in the `...(Hit|Miss)Count` metrics. However, since scanners created by compactions have `cacheBlocks` set to false, they don't affect the `...(Hit|Miss)CachingCount` metrics. Therefore, when we want to monitor the cache hit ratio specifically for client read requests connected to the HBase cluster (excluding internal operations like compactions), we need to check the `...HitCachingRatio` metric, which is what we currently monitor.
This can be easily verified with a simple test. With BucketCache enabled and an empty region server, follow these steps:
```
create 'test', 'cf'
put 'test', 'r1', 'cf:q', 'v1'
flush 'test'
put 'test', 'r2', 'cf:q', 'v2'
flush 'test'
# 1) Load one of the two blocks into the block cache
get 'test', 'r1', 'cf:q'
# 2) Observe metric changes after major compaction
major_compact 'test'
```
After step 1, all hit metrics remain at 0 (no change), while both `missCount` and `missCachingCount` increase to 2 (not entirely sure why it's 2, but that's what we observe).
After step 2, `hitCount` becomes 1, `hitCachingCount` remains 0, `missCount` becomes 3, and `missCachingCount` stays at 2.
(These metrics were observed from the L2 cache section in the region server web UI.)
This demonstrates that reads from compactions are reflected in the general hit/miss metrics but not in the Caching metrics.
> I'm not sure this is true. We do account and expose separate hit rations for L1 and L2.
For the reasons explained above, we specifically want to monitor the `(Hit|Miss)Caching` metrics rather than the general `Hit|Miss` metrics.
When `BucketCache` is enabled, `BlockCache` is used as L1 and `BucketCache` as L2, which is represented as `CombinedBlockCache` in the HBase implementation. The currently exposed `BlockCache(...Count|...Ratio)` metrics represent the combined sum of L1 and L2. While we need to monitor each cache tier separately, there are currently no `Caching` metrics available for individual cache layers. The metrics you mentioned only include `(Hit|Miss)Count` format metrics, which—as I explained above—are significantly influenced by operations like compactions and don't represent the values we're looking for.
The `Caching` metrics for each cache tier are not currently exposed via JMX (which is what this PR aims to enable), but they can be viewed in the BlockCache section of the region server web UI. As an example from our production cluster, here are the L2 metrics:
- Hits: 14,709,630
- Hits Caching: 12,340,830
- Misses: 59,908,348
- Misses Caching: 17,610,008
- Hit Ratio: 19.71%
When we calculate the Hit Caching Ratio from these values, we get 41.2%, which shows a significant difference from the 19.7% general hit ratio—demonstrating why separate caching metrics are important for accurate monitoring of client-driven cache performance.
Please let me know if anything is still unclear or if I've misunderstood something. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29731 Add L1 and L2 cache hit/miss caching metrics [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7482:
URL: https://github.com/apache/hbase/pull/7482#issuecomment-3580027050
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 34s | | master passed |
| +1 :green_heart: | compile | 1m 14s | | master passed |
| +1 :green_heart: | javadoc | 0m 42s | | master passed |
| +1 :green_heart: | shadedjars | 6m 11s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 8s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 40s | | the patch passed |
| +1 :green_heart: | shadedjars | 6m 11s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 36s | | hbase-hadoop-compat in the patch passed. |
| +1 :green_heart: | unit | 209m 15s | | hbase-server in the patch passed. |
| | | 239m 6s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7482/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7482 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 8ff34c1e2ee0 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 179f050385f630e97cb4b8825094c0810fb3fa8c |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7482/1/testReport/ |
| Max. process+thread count | 4425 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7482/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29731 Add L1 and L2 cache hit/miss caching metrics [hbase]
Posted by "wchevreuil (via GitHub)" <gi...@apache.org>.
wchevreuil merged PR #7482:
URL: https://github.com/apache/hbase/pull/7482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29731 Add L1 and L2 cache hit/miss caching metrics [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7482:
URL: https://github.com/apache/hbase/pull/7482#issuecomment-3579127004
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 17s | | master passed |
| +1 :green_heart: | compile | 3m 48s | | master passed |
| +1 :green_heart: | checkstyle | 1m 11s | | master passed |
| +1 :green_heart: | spotbugs | 2m 0s | | master passed |
| +1 :green_heart: | spotless | 0m 50s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 57s | | the patch passed |
| +1 :green_heart: | compile | 3m 43s | | the patch passed |
| +1 :green_heart: | javac | 3m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 7s | | the patch passed |
| +1 :green_heart: | spotbugs | 2m 10s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 11m 17s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.1. |
| +1 :green_heart: | spotless | 0m 44s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 19s | | The patch does not generate ASF License warnings. |
| | | 42m 14s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7482/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7482 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux d56f9de4e1c6 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 179f050385f630e97cb4b8825094c0810fb3fa8c |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 83 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7482/1/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29731 Add L1 and L2 cache hit/miss caching metrics [hbase]
Posted by "wchevreuil (via GitHub)" <gi...@apache.org>.
wchevreuil commented on PR #7482:
URL: https://github.com/apache/hbase/pull/7482#issuecomment-3581983889
Got it, so we already account these miss/hitCachingCount separately in the specific cache implementations, but we don't report those individually as the other metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org