[jira] [Commented] (HDDS-16548) [Specula] Verify recon-container-reports (Apache Ozone master @916aa1d)

["Siyao Meng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-16548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127790#comment-18127790 ] 

Siyao Meng commented on HDDS-16548:
-----------------------------------

Verification run complete. The Specula pipeline ran all phases against the pinned commit: code analysis, TLA+ specification generation, harness and trace collection, trace validation plus TLC model checking, bug confirmation with the adversarial Challenger debate, and severity classification.

h3. Run environment
{noformat}
Ozone commit: 916aa1d9949cbb10f2750a40c8704340893131a2
Specula:      v1.2.0-32-g54fe3a93 (with local changes)
Agent/model:  claude-code, Claude Opus 5.5 (1M context)
Effort:       medium
TLC limits:   28G memory, 8 workers
Total:        3h 25m agent time, 181.8M tokens, about $102 (estimate)
{noformat}

h3. Result
12 candidate findings came out of model checking (10) and code review (2), and all 12 were investigated individually.

||Disposition||Count||
|Reproduced|9|
|Masked (defect present, harm prevented by another component today)|3|

Some findings share one fix and are filed as one issue: two dead node handling findings (HDDS-17067), two Recon rebuild findings (HDDS-17061), and the health task retry loop, which is filed with the pipeline sync loop (HDDS-17071). Each issue was then validated a second time by hand against the pinned commit: a regression test was added to an existing suite and shown to fail without the fix and pass with it, the bug was confirmed still present on apache master, and the fix patch, reproduction and report were reviewed independently before filing. The three masked findings are filed as well, because each leaves Recon's (and for two of them SCM's) view wrong until the next periodic full container report. One further defect of the same kind as HDDS-17071 was found during validation and is filed as HDDS-17065.

h3. Filed
||Issue||Priority||Summary||
|HDDS-17064|Major|Full container report carried by a failed heartbeat is not resent until the next periodic report, hiding volume failure losses for up to two hours|
|HDDS-17067|Major|Registration does not refresh a known datanode's heartbeat time, so the dead node handler keeps or erases its replicas wrongly|
|HDDS-17066|Major|Recon removes the replicas of a datanode stopped for maintenance when it has not yet learned IN_MAINTENANCE from SCM|
|HDDS-17071|Major|Recon PipelineSyncTask stops after one failed cycle and ContainerHealthTask retries a failed sweep without waiting|
|HDDS-17062|Minor|ICRs put back after a failed datanode heartbeat are sent after the newer full container report that replaced them|
|HDDS-17063|Minor|ICR sent after the volume failure full container report re-adds the lost replica in Recon and SCM|
|HDDS-17060|Minor|Recon keeps a container CLOSED after SCM moves it from DELETING or DELETED back to QUASI_CLOSED|
|HDDS-17061|Minor|Recon drops the replicas of a DELETED container when it moves it back to SCM's live state|
|HDDS-17070|Minor|Recon does not follow container suppression changes made in SCM after it has added the container|
|HDDS-17069|Minor|Recon's per datanode open container count drifts from the containers that are actually open|
|HDDS-17065|Minor|Recon ContainerSizeCountTask stops for good after one failed run, freezing the container size distribution until restart|

Priorities follow the manual validation, not the pipeline's severity classes. Each issue has a proposed fix patch and a standalone reproduction test attached.

h3. Not filed
* Part of a code review finding: a replica with an out of range EC index aborting the whole health sweep. Both report paths already reject such a replica, so it cannot reach the sweep.
* Part of a code review finding: drift in Recon's replica history and heartbeat maps. Not reproduced; only the open container count part is filed (HDDS-17069).

h3. Reproduce
{code:none}
specula run --agent=claude-code --effort=medium --model='claude-opus-5-5[1m]' \
  --max-parallel=2 --enable-reviews --confirm-debate \
  --tlc-memory-limit=28G --tlc-worker-limit=8 \
  "recon-container-reports|apache/ozone|Java|Use the target-specific guidance"
{code}

h3. Caveats
* The SCM side of HDDS-17064, HDDS-17067, HDDS-17062 and HDDS-17063 is argued from the shared datanode and SCM code and was not run against SCM. The reproductions exercise the Recon endpoint.
* Several reproductions control timing or inject the fault that starts the scenario: a heartbeat RPC failure (HDDS-17062, HDDS-17064), a failed volume (HDDS-17063, HDDS-17064), a latch in the dead node handler (HDDS-17067), an SCM RPC or Recon DB failure (HDDS-17071, HDDS-17065), and a failed container add (HDDS-17061). HDDS-17066 was also run on a mini cluster with only the maintenance admin operation and a datanode shutdown.
* HDDS-17067 changes SCM behaviour: a known datanode that registers again leaves STALE or DEAD at the next health check instead of after its first heartbeat.
* HDDS-17062, HDDS-17063 and HDDS-17064 change {{StateContext}}, as does the open pull request for HDDS-16723 (apache/ozone#11423); the main code of all three applies cleanly on its head commit.
* The patches were built and tested on the pinned commit and each applies to master at d77edaa6c15. Some pairs touch the same test file and need a trivial rebase for whichever lands second: HDDS-17060, HDDS-17061 and HDDS-17070 ({{TestReconSCMContainerSyncIntegration}}), HDDS-17066 and HDDS-17067 ({{TestReconNodeManager}}), HDDS-17062 and HDDS-17064 ({{TestHeartbeatEndpointTask}}).
* The patches are proposals for review.

Generated with Specula (Claude Opus 5.5).

> [Specula] Verify recon-container-reports (Apache Ozone master @916aa1d)
> -----------------------------------------------------------------------
>
>                 Key: HDDS-16548
>                 URL: https://issues.apache.org/jira/browse/HDDS-16548
>             Project: Apache Ozone
>          Issue Type: Sub-task
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Minor
>              Labels: help-wanted, specula
>         Attachments: recon-container-reports.guidance.md
>
>
> This is a runnable Apache Ozone formal-verification target for the Specula pipeline (TLA+ model checking plus trace validation against the real code). It has not been run yet. Anyone can pick it up: run Specula against the pinned commit, then report findings on this issue. Part of umbrella HDDS-15926.
> h3. Target
> {{recon-container-reports}}. Scope and code entry points are in the attached guidance file [^recon-container-reports.guidance.md].
> h3. Run environment
> {noformat}
> Apache Ozone commit: 916aa1d9949cbb10f2750a40c8704340893131a2
> Specula:             https://github.com/specula-org/Specula (v1.1.0 or later)
> Toolchain:           JDK 21, ~28 GB RAM for TLC, an LLM coding agent (claude-code or codex adapter)
> Runtime:             hours per target; needs an agent API budget
> {noformat}
> h3. How to run
> # Clone Specula and install per its README; clone apache/ozone and check out the pinned commit above.
> # Place the attached guidance file as the target's .prompt-extra.md (see the Specula README for target layout).
> # Run:
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original --max-parallel=2 \
>   --enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8 \
>   "recon-container-reports|apache/ozone|Java|Use the target-specific .prompt-extra.md"
> {code}
> # Deliverables land under the run's {{.specula-output/}}: {{confirmed-bugs.md}}, {{bug-severity.md}}, {{summary.md}}, and {{confirmation/<id>/investigation.md}}.
> h3. Reporting results
> Comment here with the outcome (0 findings is still useful: it records the target as covered at this commit). For each confirmed bug, file a separate Ozone Bug and link it to this issue with the "Testing" link type ("Testing discovered").
> h3. Notes
> * The attached guidance file names an earlier commit ({{20e6a1c}}); use the commit above. Code entry-point line numbers in the guidance were captured at an earlier commit and may have shifted on this head; treat them as approximate. Specula re-analyzes the source, so the run does not depend on them.
> * Recon-building targets need a local Specula patch to skip Maven {{target/}} out-of-tree symlinks in {{snapshotlib._validate_source_tree}}, else the run crashes at finalize.
> * effort=medium is the tested default for large targets; effort=high can hit per-turn timeouts on the biggest ones.
> * Findings are proposals, to be confirmed by human review and testing before any fix is merged.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org