(iggy) 07/07: docs(server): rewrite the boot-recovery block for the shipped behaviour

[hg...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

hubcio pushed a commit to branch fix/torn-segment-truncate
in repository https://gitbox.apache.org/repos/asf/iggy.git

commit 689aa64cf81851f34832fdecb3d5ab01b0a0ed8d
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Mon Aug 24 09:28:45 2026 +0200

    docs(server): rewrite the boot-recovery block for the shipped behaviour
    
    The recovery paragraph in config.toml -- the only operator-facing
    description of a destructive boot path -- described behaviour the
    code no longer has: it claimed unclassifiable residue always refuses
    (the width gate is gone and the verify budget now decides), that an
    intact-index walk trusts headers unconditionally (contradicting
    batches checksum-verify), that a damaged index is rebuilt (only a
    LOST one is; damaged floors or refuses), and that refused files
    always stay in place at replica_count = 1 (the StorageSizeMismatch
    tombstone lasts one process, and zero-recoverable-byte chains
    rebuild).
    
    Rewrite it against the current head: the lost-index rebuild triggers
    and their full-rescan boot cost, the checksum gate on walk
    contradictions in both directions plus the partition_id check, the
    verify-budget refusal and its recover-empty convergence when nothing
    walked, the two tombstone exits (zero-byte rebuild, operator
    delete), and the second .fenced.N producer under a partition that
    keeps serving -- the one outcome that loses data while the partition
    stays up, previously taught backwards.
---
 core/server/config.toml | 72 +++++++++++++++++++++++++++++++++++++------------
 1 file changed, 55 insertions(+), 17 deletions(-)

diff --git a/core/server/config.toml b/core/server/config.toml
index 5b40ec0a5..2bac202b8 100644
--- a/core/server/config.toml
+++ b/core/server/config.toml
@@ -471,23 +471,61 @@ recreate_missing_state = false
 
 # At boot, segment recovery walks each partition's segments: bytes after the
 # last decodable batch of a genuinely torn tail are physically truncated from
-# the .log/.index files, and the index is rebuilt when it was damaged. Only
-# the walk of a segment whose index was lost re-checksums batches; with an
-# intact index the walk trusts batch headers (decodable, contiguous offsets),
-# and bytes before the last index entry are not re-examined at boot at all --
-# at-rest damage there surfaces on the read path via validate_checksum.
-# Damage in the middle of a segment, or trailing bytes too large or costly to
-# prove torn, is never silently truncated: the partition is refused. With
-# peer replicas the refused files are moved to a .fenced.N directory beside
-# the partition, which is rebuilt empty and refilled from a peer. With
-# replica_count = 1 there is no peer: the refused files stay at their
-# original paths, the partition is tombstoned (unrouted, never served
-# empty), and the same refusal is re-derived and re-logged on every boot
-# until an operator intervenes. Only single-replica refusals with no
-# recoverable bytes at stake (a hole from a stray file, an orphaned empty
-# segment) still rebuild through .fenced.N. The metadata WAL truncates
-# genuinely torn tails too; interior WAL damage or oversized trailing bytes
-# refuse boot instead.
+# the .log/.index files. A LOST index is rebuilt from the batches the walk
+# proves -- lost meaning absent altogether, or holding no whole 24-byte
+# entry. An index that still holds whole entries is NEVER rebuilt, only
+# floored to whole entries, or the partition is refused when its entries
+# contradict the log. Only the walk of a segment whose index was lost
+# re-checksums every batch; with an intact index the walk trusts batch
+# headers, except that a batch contradicting the walk (see below) must pass
+# its batch checksum before the contradiction is believed. Bytes before the
+# last index entry are not re-examined at boot at all -- at-rest damage
+# there surfaces on the read path via validate_checksum.
+#
+# A restore that dropped every .index therefore boots by re-reading and
+# re-checksumming every byte of every segment and rebuilding each index
+# durably, two fsyncs per segment: budget minutes, not seconds, on a
+# partition holding thousands of segments. A boot that looks hung after such
+# a restore is doing that work. Before, that same shape refused the boot
+# outright.
+#
+# Either walk refuses a batch whose base offset does not continue the chain,
+# or whose partition_id stamp is not this partition's. A contradiction whose
+# batch fails its own checksum is damage rather than data: the walk stops
+# there and the residue goes through the damage probe like any other torn
+# tail, so a bit flip in a tail batch's offset truncates instead of refusing
+# the partition, regardless of the flip's direction.
+#
+# Damage in the middle of a segment is never silently truncated: the
+# partition is refused, as is trailing residue whose verification cost
+# exhausts the probe's residue-derived work budgets while walked batches
+# exist (residue after a walk that proved nothing recovers the segment as
+# empty instead, fencing the file pair aside whole). With peer replicas the
+# refused files are moved to a .fenced.N directory beside the partition,
+# which is rebuilt empty and refilled from a peer. With replica_count = 1
+# there is no peer, so nothing is moved and nothing is rebuilt: the refused
+# files stay at their original paths, the partition is tombstoned (unrouted,
+# clients get the retriable transient status, never an empty poll that would
+# read as a healthy empty partition), and the same refusal is re-derived and
+# re-logged on every boot. The tombstone lifts in exactly two ways: a
+# chain-shape refusal whose planned chain provably holds ZERO recoverable
+# bytes rebuilds through .fenced.N at boot, and deleting the
+# stream/topic/partition removes the refused files and clears the fence, so
+# a recreate of the same ids starts clean. One refusal is not durable: a
+# length divergence found when a writer reopens a file recovery just
+# truncated clears on the next boot, so that tombstone lasts only for the
+# life of the process.
+#
+# .fenced.N has a second producer, on a partition that recovers FINE: a tail
+# segment holding bytes that decode to nothing anywhere is moved there and
+# the segment recovered empty, so the partition serves without those bytes.
+# One fresh directory per such segment, so .fenced.0, .fenced.1, ... can
+# accumulate under a healthy partition. .fenced.N alone therefore does not
+# mean a partition was refused -- grep the boot log for "refusing the
+# recovered segment chain", which names the directory and the reason.
+#
+# The metadata WAL truncates genuinely torn tails too; interior WAL damage
+# or oversized trailing bytes refuse boot instead.
 
 # Memory pool configuration
 [system.memory_pool]