You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "mazhengxuan (Jira)" <ji...@apache.org> on 2026/08/30 16:30:00 UTC
[jira] [Commented] (HBASE-29904) Tombstone-heavy regions are not prioritized for major compaction, causing severe scan latency after deletes
[ https://issues.apache.org/jira/browse/HBASE-29904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109701#comment-18109701 ]
mazhengxuan commented on HBASE-29904:
-------------------------------------
Hi, I'd like to work on this issue.
I checked the current master and the related changes in HBASE-29039 and HBASE-30036. I will first reproduce the problem with the current code, then look into adding store-level delete-marker metrics and an optional threshold for triggering major compaction, reusing the existing compaction controls.
If this approach sounds reasonable, I‘ll assign the issue to me.
Thanks!
> Tombstone-heavy regions are not prioritized for major compaction, causing severe scan latency after deletes
> -----------------------------------------------------------------------------------------------------------
>
> Key: HBASE-29904
> URL: https://issues.apache.org/jira/browse/HBASE-29904
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 2.5.10
> Reporter: Emil Kleszcz
> Priority: Major
>
> After large delete workloads, some regions accumulate very high numbers of tombstones. Until a major compaction runs, these regions suffer severe scan latency degradation from milliseconds to tens of seconds or even minutes.
> HBase currently:
> * does not expose tombstone density per region/store, and
> * does not prioritize major compactions based on tombstone impact.
> As a result, periodic major compactions may miss the worst regions for days or weeks, even though user-facing queries are already severely affected.
> Observed behavior:
> * Before deletes or after major compaction: ~50–200 ms
> * After deletes, before major compaction: 10–120 seconds
> * After manual major compaction on the affected region: back to milliseconds
> The slowdown is caused by scanning and filtering large volumes of delete markers (tombstones), not by storefile count or region size. Minor compactions do not resolve the issue. The issue is not present when _RAW_ option is set to _true_.
> *Current limitations*
> * No metric to identify tombstone-heavy regions.
> * Compaction selection is driven by file count / size, or last compaction time, not delete density.
> * Operators must manually correlate slow queries with regions and trigger targeted major compactions.
> * This risks both over-scheduling majors and missing the worst regions.
> *Requested improvement*
> * Expose tombstone-related metrics (e.g. delete marker count or ratio) at region/store level.
> * Allow compaction scheduling to prioritize regions with high tombstone density or reclaimable space.
> * Provide controls to rate-limit and restrict such tombstone-driven major compactions (e.g. off-peak only).
> *Expected benefit*
> * Predictable scan latency after deletes.
> * Faster reclamation of deleted data.
> * Reduced need for operator-driven major compactions prone to over-scheduling.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)