[jira] [Commented] (CASSANDRA-21485) AutoRepair metric BytesAlreadyRepaired can exceed TotalBytesToRepair due to memtable data not included in total estimation

["George (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-21485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18119809#comment-18119809 ] 

George commented on CASSANDRA-21485:
------------------------------------

Patch available for review: https://github.com/apache/cassandra/pull/5216

The plan now snapshots the memtable estimate once per table, shares it across token ranges, and reuses those saved estimates when generating repair assignments. Flushes or subsequent writes therefore do not change the estimate between statistics collection and assignment generation. Fixed-split assignments also preserve the plan's total, including integer remainders and zero-byte assignments.

Regression coverage includes all three repair types, flushes and writes between planning and assignment generation, multiple ranges and tables, both fixed-split modes, rounding, empty tables, and SSTable/memtable coexistence. The scheduler statistics test now starts repair only after its setup and flushes finish.

Validation on trunk, commit e70566a758697ddb58cd3a33f3f34368a658c365 (macOS, JDK 17.0.19, Ant 1.10.13):
* .build/check-code.sh -s: passed (build, RAT and Checkstyle).
* AutoRepairStateTest: 99 passed.
* RepairTokenRangeSplitterTest: 50 passed.
* FixedSplitTokenRangeSplitter tests: 162 passed.
* AutoRepairSchedulerStatsNoVNodesTest and AutoRepairSchedulerStatsVNodesTest: 1 passed each.

Total: 311 unit tests and 2 JVM distributed tests passed. Exact commands are in the PR description.

Community pre-commit CI has not been run; I do not currently have access configured. Could a reviewer or committer help run it and attach the ci_summary and results_details artifacts here?

There are no configuration, wire-format or on-disk format changes. The metrics remain estimates of repair work, rather than measured transfer sizes.

> AutoRepair metric BytesAlreadyRepaired can exceed TotalBytesToRepair due to memtable data not included in total estimation
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-21485
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-21485
>             Project: Apache Cassandra
>          Issue Type: Task
>            Reporter: Paulo Motta
>            Priority: Normal
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> h2. Summary
> The {{TotalBytesToRepair}} metric does not account for memtable data, causing {{BytesAlreadyRepaired}} to exceed {{TotalBytesToRepair}} (progress > 100%) after a repair turn completes.
> h2. Root Cause
> {{TotalBytesToRepair}} is computed in {{AutoRepairState.updateRepairScheduleStatistics()}} by summing {{KeyspaceRepairPlan.getEstimatedBytes()}}, which only considers on-disk SSTable sizes via {{SizeEstimate.sizeForRepair}}.
> However, during repair assignment generation, {{RepairTokenRangeSplitter.getRepairAssignments()}} (line 522-527) discovers tables with no SSTables but active memtable data and creates assignments with {{estimatedBytes = memtableSize}}. These bytes are then accumulated in {{bytesAlreadyRepaired}} but were never counted in the total.
> h2. Code References
> * {{AutoRepairState.java:117-119}} - computes {{totalBytesToRepair}} from plan (SSTable-only)
> * {{RepairTokenRangeSplitter.java:522-527}} - discovers memtable data and assigns {{memtableSize}} as estimated bytes
> * {{AutoRepair.java:401}} - accumulates {{bytesAlreadyRepaired}} including memtable-based assignments
> * {{AutoRepairUtils.java:1391}} - {{sizeForRepair}} is derived from SSTable sizes only
> h2. Impact
> * The {{BytesAlreadyRepaired}} and {{TotalBytesToRepair}} JMX metrics report inconsistent values (repaired > total)
> * Progress monitoring dashboards show values above 100% which is confusing for operators
> * The repair itself is not affected - all data (including memtable) is correctly repaired
> h2. Suggested Fix
> Include memtable sizes in the total estimation. When computing {{totalBytesToRepair}}, also check for live memtable data on tables that have {{sizeForRepair == 0}}, mirroring the logic in the splitter. This could be done in {{calcTotalBytesToBeRepaired()}} or {{updateRepairScheduleStatistics()}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org