You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/09/16 03:13:00 UTC
[jira] [Updated] (HDDS-16442) After OM upgrade, pre-existing DONE snapshot diff jobs are permanently unreadable
[ https://issues.apache.org/jira/browse/HDDS-16442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siyao Meng updated HDDS-16442:
------------------------------
Description:
h3. Finding
After a routine OM upgrade, every pre-existing DONE snapshot-diff job deserializes with a null largestEntryKey, so the first client read of the report trips the integrity check and hits an uncaught NPE (wrong-key updateJobStatus) instead of the designed retriable error; the report is intact yet permanently unreadable and the job is stuck DONE (not resubmittable) until the retention TTL GC, affecting all old jobs broadly rather than via a rare race.
h3. Classification
* Verdict: REPRODUCED
* Severity: Critical
* Source: Specula TLA+ model checking and confirmation debate, finding CR-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-snapshot-diff-jobs
Guidance: campaigns/ozone-9fbf9ee/targets/009-om-snapshot-diff-jobs/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8 \
"om-snapshot-diff-jobs|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16430 (Specula TLA+ verification effort). The TLA+ specification, counterexample, and confirmation debate live in the Specula run artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
Precondition (reachable through a routine upgrade): a snapshot diff job for (fromSnapshot, toSnapshot) completed and was persisted as DONE by an OM build that predates the largestEntryKey field on SnapshotDiffJob. After the OM is upgraded, that job deserializes with largestEntryKey == null and stays in the active job table (loadJobsOnStartUp leaves DONE jobs untouched), with its report rows fully intact.
1. Client reads the completed diff: getSnapshotDiffReport routes the DONE job to createPageResponse.
2. On the last page, checkReportsIntegrity sees largestEntryKey == null and treats it as an integrity failure.
3. The recovery path calls updateJobStatus(diffJob.getJobId(), DONE, FAILED). The job table is keyed by the dedup jobKey (fromSnapshotId + DELIMITER + toSnapshotId), not by the per-generation jobId, so the lookup returns null and dereferencing its status throws NullPointerException.
4. The designed retriable IOException ("Report integrity check failed. Retry after: ...") is never thrown, the DONE to FAILED transition never lands, and the job is stuck DONE (not a resubmittable status), so the client cannot recover until the report-retention TTL GC. Every read repeats the NPE.
The same NPE occurs for any genuine last-page integrity mismatch (non-null largestEntryKey that disagrees with the report tail), since the recovery call always passes the wrong key.
h3. Root cause
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java: checkReportsIntegrity called updateJobStatus(diffJob.getJobId(), DONE, FAILED), passing the jobId where updateJobStatus expects the dedup jobKey (it does snapDiffJobTable.get(jobKey)). Two problems compound: (1) a null largestEntryKey (legacy DONE job) is treated as a failure instead of a "no stored last entry" value, and (2) updateJobStatus has no null guard, so the wrong-key lookup NPEs on getStatus().
h3. Suggested fix
Thread the dedup jobKey down through createPageResponse into checkReportsIntegrity and pass it to updateJobStatus. Treat a null largestEntryKey on a DONE job as a legacy value and skip the last-page integrity assertion so upgraded jobs stay readable. Null-guard updateJobStatus so a missing row logs and returns instead of throwing. Regression tests cover both the legacy-null (stays readable) and genuine-mismatch (flips to FAILED via jobKey, throws the retriable IOException) cases. See [^HDDS-16442.001.patch].
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full test, and review before merge.
was:
h3. Finding
After a routine OM upgrade, every pre-existing DONE snapshot-diff job deserializes with a null largestEntryKey, so the first client read of the report trips the integrity check and hits an uncaught NPE (wrong-key updateJobStatus) instead of the designed retriable error; the report is intact yet permanently unreadable and the job is stuck DONE (not resubmittable) until the retention TTL GC, affecting all old jobs broadly rather than via a rare race.
h3. Classification
* Verdict: REPRODUCED
* Severity: Critical
* Source: Specula TLA+ model checking and confirmation debate, finding CR-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-snapshot-diff-jobs
Guidance: campaigns/ozone-9fbf9ee/targets/009-om-snapshot-diff-jobs/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8 \
"om-snapshot-diff-jobs|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16430 (Specula TLA+ verification effort). The TLA+ specification, counterexample, and confirmation debate live in the Specula run artifacts.
Generated with Specula (Claude Opus 4.8).
> After OM upgrade, pre-existing DONE snapshot diff jobs are permanently unreadable
> ---------------------------------------------------------------------------------
>
> Key: HDDS-16442
> URL: https://issues.apache.org/jira/browse/HDDS-16442
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Critical
> Attachments: HDDS-16442.001.patch
>
>
> h3. Finding
> After a routine OM upgrade, every pre-existing DONE snapshot-diff job deserializes with a null largestEntryKey, so the first client read of the report trips the integrity check and hits an uncaught NPE (wrong-key updateJobStatus) instead of the designed retriable error; the report is intact yet permanently unreadable and the job is stuck DONE (not resubmittable) until the retention TTL GC, affecting all old jobs broadly rather than via a rare race.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: Critical
> * Source: Specula TLA+ model checking and confirmation debate, finding CR-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: om-snapshot-diff-jobs
> Guidance: campaigns/ozone-9fbf9ee/targets/009-om-snapshot-diff-jobs/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8 \
> "om-snapshot-diff-jobs|apache/ozone|Java|Use the target-specific .prompt-extra.md"
> {code}
> Discovered under HDDS-16430 (Specula TLA+ verification effort). The TLA+ specification, counterexample, and confirmation debate live in the Specula run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> Precondition (reachable through a routine upgrade): a snapshot diff job for (fromSnapshot, toSnapshot) completed and was persisted as DONE by an OM build that predates the largestEntryKey field on SnapshotDiffJob. After the OM is upgraded, that job deserializes with largestEntryKey == null and stays in the active job table (loadJobsOnStartUp leaves DONE jobs untouched), with its report rows fully intact.
> 1. Client reads the completed diff: getSnapshotDiffReport routes the DONE job to createPageResponse.
> 2. On the last page, checkReportsIntegrity sees largestEntryKey == null and treats it as an integrity failure.
> 3. The recovery path calls updateJobStatus(diffJob.getJobId(), DONE, FAILED). The job table is keyed by the dedup jobKey (fromSnapshotId + DELIMITER + toSnapshotId), not by the per-generation jobId, so the lookup returns null and dereferencing its status throws NullPointerException.
> 4. The designed retriable IOException ("Report integrity check failed. Retry after: ...") is never thrown, the DONE to FAILED transition never lands, and the job is stuck DONE (not a resubmittable status), so the client cannot recover until the report-retention TTL GC. Every read repeats the NPE.
> The same NPE occurs for any genuine last-page integrity mismatch (non-null largestEntryKey that disagrees with the report tail), since the recovery call always passes the wrong key.
> h3. Root cause
> hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java: checkReportsIntegrity called updateJobStatus(diffJob.getJobId(), DONE, FAILED), passing the jobId where updateJobStatus expects the dedup jobKey (it does snapDiffJobTable.get(jobKey)). Two problems compound: (1) a null largestEntryKey (legacy DONE job) is treated as a failure instead of a "no stored last entry" value, and (2) updateJobStatus has no null guard, so the wrong-key lookup NPEs on getStatus().
> h3. Suggested fix
> Thread the dedup jobKey down through createPageResponse into checkReportsIntegrity and pass it to updateJobStatus. Treat a null largestEntryKey on a DONE job as a legacy value and skip the last-page integrity assertion so upgraded jobs stay readable. Null-guard updateJobStatus so a missing row logs and returns instead of throwing. Regression tests cover both the legacy-null (stays readable) and genuine-mismatch (flips to FAILED via jobKey, throws the retriable IOException) cases. See [^HDDS-16442.001.patch].
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org