[jira] [Commented] (HDDS-16176) EC replicated size is computed per key on commit but per block on delete

["Chia-Chuan Ho (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18108231#comment-18108231 ] 

Chia-Chuan Ho commented on HDDS-16176:
--------------------------------------

Thanks [~chihsuan] for the detailed report.
 
I confirmed the inconsistency and agree that quota accounting should use the per-block-group value, since each EC block group is encoded independently.
 
The proposed fix is to: * calculate committed quota from each block in the newly committed location group, for both regular and FSO keys;
 * apply the same calculation to MPU part commit, overwrite, abort, expired abort, and completion cleanup;
 * use the same block-based calculation when quota repair rebuilds usedBytes and deleted-key snapshotUsedBytes;
 * keep OmKeyInfo.getReplicatedSize() unchanged, since it also has non-quota callers;
 * avoid clamping OmBucketInfo.usedBytes at zero, as that could hide the inconsistency and discard usage belonging to other keys.

 
Existing EC buckets would need quota repair after upgrading, because OM cannot distinguish keys charged with the historical per-key formula from keys charged with the corrected formula without adding persistent versioning or an upgrade migration.
 
One wording detail: the per-block value is never smaller than the per-key value, but it is not always strictly larger. They are equal when block-group boundaries align with complete EC stripes; the mismatch occurs when multiple block groups independently incur partial-stripe parity.
 
Does this scope and migration approach look right to you, especially including the MPU paths and requiring quota repair for existing buckets?  Thanks!

> EC replicated size is computed per key on commit but per block on delete
> ------------------------------------------------------------------------
>
>                 Key: HDDS-16176
>                 URL: https://issues.apache.org/jira/browse/HDDS-16176
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: OM
>            Reporter: Chi-Hsuan Huang
>            Priority: Major
>
> h3. Problem
> For an EC key whose data spans more than one block, OM charges quota using a different formula than it refunds with. Commit charges {{QuotaUtil.getReplicatedSize\(dataSize, repConfig\)}}, converting the whole key at once. Delete refunds {{OMKeyRequest.sumBlockLengths\(omKeyInfo\)}}, converting each block on its own. EC parity is added per block, so the two do not agree, and the refund is always the larger of the two.
> {{OmBucketInfo.decrUsedBytes}} does not clamp at zero, so deleting such keys can drive a bucket's {{usedBytes}} negative.
> h3. Example
> With {{rs\-3\-2\-1024}}, the data stripe is 3 \* 1024 bytes. A key holding two blocks of 1024 bytes each:
> * per key: {{getReplicatedSize\(2048\)}} = 2048 \+ 1024 \* 2 parity = 4096, which is what commit charges
> * per block: \(1024 \+ 1024 \* 2 parity\) \* 2 = 6144, which is what delete refunds
> h3. Which value describes the stored bytes
> The per\-block value does. EC encodes each block group independently, so each block carries its own partial\-stripe padding:
> * {{QuotaUtil.getReplicatedSize}} is documented as "Number of bytes required to store the dataSize with replication".
> * {{KeyManagerImpl.java:852\-859}}, which reports reclaimed space to SCM, computes both {{getReplicatedSize}} and {{getSizePerReplica}} per block, and passes {{OMKeyRequest.sumBlockLengths\(info\)}} as the quota released.
> * {{hadoop\-hdds/docs/content/feature/Quota.md}} describes the space quota as the storage space a bucket may use, counted after replication.
> RATIS replication is linear, so the two formulas agree there and only EC is affected.
> h3. Also affected: snapshotUsedBytes in quota repair
> {{QuotaRepairTask.recalculateDeletedKeyUsages}} \({{QuotaRepairTask.java:513}}\) totals the deleted\-key table with {{RepeatedOmKeyInfo.getTotalSize\(\)}}, which converts each key as a whole, while {{OMKeyDeleteRequest.java:165}} moves the per\-block {{sumBlockLengths\(omKeyInfo\)}} into {{snapshotUsedBytes}}. The same divergence therefore applies to that counter, and the formula chosen here should be applied there too. That total is wrong for a second, independent reason as well, tracked in HDDS\-16184.
> h3. Impact
> Space quota accounting for EC buckets is internally inconsistent. A bucket can report a lower {{usedBytes}} than the space its keys occupy, and repeated create/delete cycles can push {{usedBytes}} below zero.
> h3. Notes
> Relates to HDDS\-16127, which changes quota repair to recount keys retaining more than one version by converting each version once, matching the commit path. That patch deliberately does not adopt the per\-block conversion, so that repair reproduces what commit charged rather than pre\-empting the decision here.
> Pinned source commit 4766aa8609. Analysis assisted by AI tooling \(Claude Code, Opus 5\).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org