You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/10/15 11:29:00 UTC
[jira] [Created] (HDDS-17048) Recon NSSummary task fails with NullPointerException on a key event whose bucket is already deleted in the Recon OM DB, forcing a full rebuild
Siyao Meng created HDDS-17048:
---------------------------------
Summary: Recon NSSummary task fails with NullPointerException on a key event whose bucket is already deleted in the Recon OM DB, forcing a full rebuild
Key: HDDS-17048
URL: https://issues.apache.org/jira/browse/HDDS-17048
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-5-skip-key-events-of-deleted-bucket.patch, TestBugMC5DeletedBucketKeyEvent.java
h3. Mechanism
Recon commits each fetched OM delta to its OM DB and then buffers the events for asynchronous processing, so the Recon OM DB can be ahead of the batch being processed. {{NSSummaryTaskWithOBS.processWithOBS}} and {{NSSummaryTaskWithLegacy.isBucketLayoutValid}} resolve the bucket of every keyTable event with {{NSSummaryTaskDbEventHandler.lookupBucketCached}}, which falls back to that DB when the bucket is not cached, and then call {{getBucketLayout()}} on the result without a null check. When the bucket was deleted in the same delta, or in a later one applied before the batch is processed, the lookup returns null and both sub-tasks throw a NullPointerException. The reprocess path ({{NSSummaryTaskWithOBS.processKeyTableInParallel}}) already skips a null bucket.
A cached bucket does not avoid it. Events of one OM double buffer flush are grouped per table, and bucketTable events come before keyTable events. When the last key delete and the bucket delete are in one flush, Recon gets the bucketTable DELETE first, which drops the cached entry, and the key event then looks the bucket up in the DB.
The NPE fails the whole batch, the controller's retry fails the same way, and the controller sets its task failure flag. The OM sync loop then queues a {{TASK_FAILURES}} reinitialization, a full rebuild of every Recon OM task. Until the rebuild is queued {{consumeOMEvents}} drops new event batches, and later batches wait behind the rebuild. The rebuild produces correct data, so nothing stays wrong; the cost is the full rebuild and a stale window for all Recon OM tasks.
h3. Trigger
A key event of an OBS or LEGACY bucket is processed after the bucket is deleted in the Recon OM DB, for example:
# A client deletes the last key of a bucket and then the bucket, and both are applied in one OM double buffer flush, which back to back requests such as {{ozone sh bucket delete -r}} or {{aws s3 rb --force}} can do. This fails whether or not the bucket was cached.
# Or the key events and the bucket delete are in different flushes, the bucket delete reaches the Recon OM DB before the key events are processed (the default fetch interval is 5s, longer when event processing lags), and the sub-tasks have not cached the bucket, for example a bucket created in the same window or one with no key activity since Recon started.
The unchecked {{getBucketLayout()}} call exists since 1.3.0 for LEGACY buckets (HDDS-7121) and since 1.4.1 for OBS buckets (HDDS-7810); those versions looked the bucket up for every event without a cache. The bucket cache added in 2.2.0 (HDDS-15335) did not change it.
h3. Impact
* Every occurrence forces a full rebuild of all Recon OM tasks and leaves their data stale until it completes. The other events of the failing batch (for other buckets) are applied only by that rebuild. On a large namespace the rebuild is long, and frequent bucket cleanup can keep Recon rebuilding.
* No lasting wrong data is expected, since the rebuild recomputes the summary from the Recon OM DB (argued from the code; the attached test stops at the task failure flag).
h3. Reproduction
PASS on unmodified source, deterministic, no fault injection. [^TestBugMC5DeletedBucketKeyEvent.java] ({{hadoop-ozone/recon}}) writes real OM DB batches (a key in a live bucket, delete of the last key of a second bucket, delete of that bucket), serves them through {{RDBStore.getUpdatesSince}}, applies them to the Recon OM DB as {{OzoneManagerServiceProviderImpl}} does and hands the events to a real {{ReconTaskControllerImpl}} running {{NSSummaryTask}}. Only the task status bookkeeping is mocked. It shows {{hasTasksFailed() = true}}, the live bucket still at 1 file instead of 2, and the {{getBucketLayout()}} NPE from both sub-tasks on process and retry. A passing test means the defect is present.
h3. Patch
[^MC-5-skip-key-events-of-deleted-bucket.patch], against 0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at 2264d4d48ac.
A key event whose bucket is no longer in the Recon OM DB is skipped in both sub-tasks, as the reprocess path already does. The bucket and its NSSummary entry are unreachable after the delete, so nothing is lost. Covered by {{TestNSSummaryTask.TestProcessParallelMixedBatch#testKeyEventOfDeletedBucketDoesNotFailBatch}} (bucket not cached) and {{#testKeyEventOfDeletedCachedBucketDoesNotFailBatch}} (cached bucket, bucketTable DELETE and keyTable DELETE in one batch), which both fail without the change. With it, the NSSummary task suites pass (54 tests) and checkstyle is clean. With the patch the reproduction fails because the batch succeeds and the live bucket counts both keys.
Related: HDDS-10855 covered the same root cause for the Legacy parent directory lookup ({{setKeyParentID}}) and only changed its messages; that path still fails a batch when a parent directory key is deleted in the same delta and is not changed here.
Found by TLA+ model checking and code review of the Recon namespace summary and task sync paths under HDDS-15926, on commit 0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org