[jira] [Updated] (HDDS-17050) Recon task cursor advances past a batch that failed both attempts, so a restart soon after the failure can lose that batch

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-17050:
------------------------------
    Attachment: TestBugMC7FailedBatchLostAcrossRestart.java

> Recon task cursor advances past a batch that failed both attempts, so a restart soon after the failure can lose that batch
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17050
>                 URL: https://issues.apache.org/jira/browse/HDDS-17050
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Minor
>         Attachments: MC-7-skip-batches-behind-failed-batch-until-reinit.patch, TestBugMC7FailedBatchLostAcrossRestart.java
>
>
> h3. Mechanism
> When a delta batch fails both attempts in {{ReconTaskControllerImpl.processOMUpdateBatch}}, the controller only sets the in memory {{tasksFailed}} flag; the next sync iteration turns it into a reinitialization. Batches that are already in the event buffer behind the failed one are still processed, and for each task that succeeds {{processTasks}} saves {{lastUpdatedSeqNumber}} to the task status table. The cursor of the failed task therefore reaches the cursor of {{OmDeltaRequest}}. If Recon restarts before the reinitialization has finished and before the sync has fetched any new OM update, the flag and the queued reinitialization are gone, and the startup check in {{OzoneManagerServiceProviderImpl.start}}, which rebuilds only tasks whose cursor is below that of {{OmDeltaRequest}}, finds nothing to do. The failed batch is never applied.
> The same happens when a manual rebuild drains a buffered batch and the rebuild's reprocess step then fails, because that failure also only sets {{tasksFailed}} (code reading, not run). A rebuild that fails in its DB swap sets no flag at all, because {{reInitializeTasks}} still reports success; HDDS-17047 makes it report the failure.
> h3. Trigger
> # The sync buffers two batches before the event thread gets to the first, as when it fetches several deltas in one run.
> # A task fails the first batch on both attempts and succeeds on the second batch. The failure has to be tied to the first batch; a Recon RocksDB I/O error closes the DB, so the second batch would fail too. One such failure is HDDS-17048: {{NSSummaryTask}} fails with a NullPointerException, on both attempts, on a key event whose bucket was deleted later in the same delta, while a later batch about other buckets succeeds (code reading).
> # Recon restarts after the event thread has processed the buffered batches and before the sync fetches any new OM update. A non empty fetch after the failure moves the {{OmDeltaRequest}} cursor past the task cursors, because the batch is refused or drained, and the startup check then rebuilds. On a cluster with ongoing writes this leaves about one sync interval (5s by default). Without new writes, it lasts until the queued reinitialization has completed.
> h3. Impact
> The changes of the failed batch are missing from that task's derived data for good: for NSSummary the namespace summary and du APIs report too few files and bytes. Later syncs and restarts do not notice it; a manual rebuild repairs it. Without a restart the reinitialization repairs it as designed.
> h3. Reproduction
> PASS, deterministic, unmodified source. [^TestBugMC7FailedBatchLostAcrossRestart.java] ({{hadoop-ozone/recon}}) uses a real OM RocksDB that serves its write ahead log through a mocked {{getDBUpdates}}, and the real {{OzoneManagerServiceProviderImpl}}, {{ReconTaskControllerImpl}}, {{NSSummaryTask}} and task status table. {{NSSummaryTask.process}} returns a failed result for one batch on both attempts, and its first attempt is held until the next batch is buffered. The restart stops the provider and controller and starts new ones on the same databases. Output before the restart: {{tasksFailed}} true, NSSummaryTask cursor 5, OmDeltaRequest cursor 5, OM has 3 files under the bucket and NSSummary 2. After the restart and a sync NSSummary still reports 2; a rebuild gives 3. A passing test means the defect is present.
> h3. Patch
> [^MC-7-skip-batches-behind-failed-batch-until-reinit.patch], against 0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at 2264d4d48ac.
> {{processOMUpdateBatch}} does not run the tasks while {{tasksFailed}} is set. In that state {{consumeOMEvents}} already refuses new batches, and queueing the reinitialization drains the buffer, clears the flag and takes the checkpoint the rebuild uses, which contains every batch applied to the Recon OM DB, so the skipped batches would be discarded by it anyway. Not processing them keeps the failed task's cursor behind, and the existing startup check rebuilds after a restart. Without a restart nothing changes, except that until the reinitialization is queued (normally the next sync iteration) the tasks that did not fail also stop applying buffered batches, which the reinitialization rebuilds.
> Covered by {{testBatchesBehindFailedBatchDoNotAdvanceTaskCursor}} in the existing {{TestReconTaskControllerImpl}}, which fails without the change ({{expected: 0L but was: 200L}}). With the change the Recon tasks suites and {{TestOzoneManagerServiceProviderImpl}} (155 tests) pass and checkstyle is clean. With the patch the reproduction fails at its undercount assertion: the NSSummaryTask cursor stays at 3, the restart rebuilds the tasks and NSSummary reports 3.
> A related case, a task whose {{process}} throws instead of returning a failed result and so never sets {{tasksFailed}}, is HDDS-17040.
> Found by TLA+ model checking and code review of the Recon namespace summary and task sync paths under HDDS-15926, on commit 0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. Related but not duplicates: HDDS-5146 (a failure between the Recon OM DB update and the task hand off loses the batch) and HDDS-16463 (sync the Recon DB before the cursor advances). The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org