[jira] [Created] (HBASE-30459) Add an option to account compaction throughput control by actual output (post-compression) bytes

["Jeongmin Kim (Jira)" <ji...@apache.org>]

Jeongmin Kim created HBASE-30459:
------------------------------------

             Summary: Add an option to account compaction throughput control by actual output (post-compression) bytes
                 Key: HBASE-30459
                 URL: https://issues.apache.org/jira/browse/HBASE-30459
             Project: HBase
          Issue Type: Improvement
          Components: Compaction
            Reporter: Jeongmin Kim


Compaction throughput limits are consumed in {{Compactor#performCompaction}} by {{cell.getSerializedSize()}} — the pre-encoding, pre-compression size of each cell — while the bytes actually written to disk are post-DATA_BLOCK_ENCODING / post-compression. Under the same limit, a store whose data encodes/compresses R times runs its compactions at roughly 1/R of the configured disk write rate.

This works against both goals of throughput control: keeping compactions inside a predictable resource envelope, and letting them actually use the resources that envelope grants. The better the encoding and compression work, the more of the budget is charged for bytes that never reach the disk, and the compaction sleeps instead of writing. In practice — we observed this in production — a compaction of a highly compressible store spends most of its wall time in throttle sleep while the disk stays nearly idle: even a compaction that is small on disk takes a long time, occupies a longCompactions thread throughout, and the compaction queue backs up. And since limits are per-RegionServer (there is no per-table or per-family limit), stores holding incompressible payloads consume the same budget 1:1, so mixed workloads additionally become unfair between families.

Proposal: an opt-in configuration {{hbase.hstore.compaction.throughput.control.by.output}} (default {{{}false{}}}, current behavior unchanged). When enabled and the sink exposes its output position, {{performCompaction}} calls {{ThroughputController#control}} with the delta of the writer's output position after each cell-batch append, instead of the cells' serialized sizes. Notes on the implementation we have been running:
 * {{StoreFileWriter#getPos}} also includes the historical file writer when {{{}hbase.enable.historical.compaction.files=true{}}}, so the whole disk write load is accounted; {{AbstractMultiFileWriter}} gains a {{getPos}} summing its lower writers (stripe / date-tiered compactors).
 * The position only advances when a block is encoded/compressed and flushed to the output stream, so the controller sees actual disk bytes. The control check interval ({{{}controlPerSize{}}}, by default the throughput lower bound) is much larger than block sizes (32K ~ 128K), so block-granularity jumps are absorbed by the accounting.
 * Progress accounting, the shipped() cadence and {{CloseChecker}} stay on cell serialized sizes — only the throughput accounting unit changes.
 * Sinks that do not expose an output position (e.g. {{{}DefaultMobStoreCompactor{}}}, which overrides {{performCompaction}} anyway) keep the existing accounting; an unexpected sink falls back with a warn log.
 * Bytes written at close time (remaining inline chunks, root index, file info, trailer) are not accounted — the existing accounting does not see them either.
 * Input-side load (reading and decompressing the source storefiles) is not accounted; for major compactions input ≈ output on disk, so the practical gap is small.
 * A side benefit: the controller's finish log ("... average throughput is ...") then reports disk MB/s, which is directly comparable with device throughput.

With the option enabled, the controller charges only what is actually written: compactions of compressible stores can use the full configured disk budget, so their duration becomes proportional to their on-disk size, and stores with incompressible payloads behave exactly as before — output accounting only differs where encoding/compression does. The implementation has been running in production with no regressions observed.

A PR against master follows.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)