[I] addEntry succeeds on replacement bookies after ensemble recovery, but entry is silently dropped — lh.close() fails with BKMetadataVersionException [bookkeeper]

["Shawyeok (via GitHub)" <gi...@apache.org>]

Shawyeok opened a new issue, #4812:
URL: https://github.com/apache/bookkeeper/issues/4812

   ## Summary
   
   When an open ledger's ensemble bookies are decommissioned and replaced with fresh bookies at the same network addresses, the original client's subsequent `addEntry` call **returns success** (entry ID assigned, LAC advanced), yet the entry is **permanently lost**: the recovery process had already closed the ledger at a lower `lastEntry` in ZK metadata, and `lh.close()` then fails with `BKMetadataVersionException`. The client receives no `BKLedgerFencedException`; the data loss is silent.
   
   ## Environment
   
   | | |
   |---|---|
   | BookKeeper version | 4.17.3 (bundled in Apache Pulsar 4.0.10) |
   | Also reproduced on | BK 4.14.x (bundled in Pulsar 3.0.x) |
   | ZooKeeper | embedded in `apachepulsar/pulsar:4.0.10` |
   | Java | OpenJDK 21 |
   
   ## Steps to Reproduce
   
   **Cluster setup**
   
   - 1 ZooKeeper node
   - 3 bookies (`bk1`, `bk2`, `bk3`) on the same Docker bridge network
   
   **Client**
   
   Opens a ledger with `E=2, WQ=2, AQ=2`. The ensemble is assigned two of the three bookies (say `bk2` and `bk3`); `bk1` is the spare.
   
   Writes `entry-0`, then blocks waiting for a proceed signal.
   
   **Decommission + replace (one at a time)**
   
   Because E=2/WQ=2 with only 3 bookies leaves one spare, the order must be interleaved:
   
   1. **D1** — `docker stop bk2` → `bookkeeper shell decommissionbookie -bookieid <bk2-ip>:3181`
      The ReplicationWorker copies bk2's fragments to bk1 (the spare) and deletes bk2's ZK cookie.
   
   2. **E1** — Start a fresh `bk2-new` at the **exact same IP** as the original `bk2`.
   
   3. **D2** — `docker stop bk3` → `bookkeeper shell decommissionbookie -bookieid <bk3-ip>:3181`
      During this step the ReplicationWorker on bk1 detects ledger 0 as under-replicated. After `openLedgerRereplicationGracePeriod` (30 s default) it performs recovery-mode open: fences the ledger, reads up to the last confirmed entry (`lastEntry=0`), closes it in ZK metadata. ZK version is bumped.
   
   4. **E2** — Start a fresh `bk3-new` at the **exact same IP** as the original `bk3`.
   
   **Proceed signal**
   
   After both replacements are registered, the client's proceed sentinel is created.
   
   ```
   addEntry("entry-1")  →  id=1, LAC=1   ← SUCCESS, no exception
   lh.close()           →  BKMetadataVersionException: Bad ledger metadata version
   ```
   
   ## Observed Behaviour
   
   ```
   [client] entry-1 id=1  LAC=1
   [client] ERROR lh.close(): BKMetadataVersionException: Bad ledger metadata version
   ```
   
   Log from the recovery side (during D2):
   
   ```
   INFO  LedgerFragmentReplicator - Updated ZK to point ledger fragments from old bookies
         to new bookies: {<bk3-ip>:3181=<bk2-new-ip>:3181}
   ```
   
   And from the original client's close path:
   
   ```
   ERROR Metadata conflict when closing ledger 0. Another client may have recovered
         the ledger while there were writes outstanding.
         (local lastEntry:1 length:14)  (metadata lastEntry:0 length:7)
   ```
   
   ## Expected Behaviour
   
   The original client should receive `BKLedgerFencedException` (or equivalent) on the `addEntry` call — **not** a success acknowledgement — because the ledger was already fenced and closed by the recovery process before the write was attempted.
   
   Alternatively, if `addEntry` is allowed to succeed, `lh.close()` should not silently discard the entry: it should either succeed (reconcile the metadata) or propagate a clear error that signals the committed entry can no longer be confirmed.
   
   ## Root Cause Analysis
   
   BookKeeper's fencing protocol works by sending `fenceLedger` RPCs to the **current ensemble** bookies. Those bookies record the fence in memory and reject future writes with `BKLedgerFencedException`. This relies on the **same** bookie processes being present at the time of both the fence and any subsequent write attempt.
   
   When a bookie is replaced by a **fresh process at the same IP and port**, the new bookie has no in-memory fence state for any previously open ledger. The original client — whose `LedgerHandle` still holds a valid connection to the (now new) bookies — sends `addEntry` RPCs which the new bookies accept without complaint, because from their perspective this ledger has never been fenced.
   
   The result is that `addEntry` completes with a success acknowledgement from `AQ` (=2) replicas, advancing `LAC` to 1. The entry is physically written to the new bookies' journals. But since ZK metadata records `lastEntry=0` (set during recovery before the writer wrote), the entry will never be visible to any future reader and is functionally lost.
   
   The `BKMetadataVersionException` from `lh.close()` is the only signal the writer receives, and it arrives **after** `addEntry` has already confirmed the write — too late for the application to react.
   
   ## Impact
   
   - **Silent data loss**: writes that receive a successful `addEntry` acknowledgement can be permanently dropped with no indication to the writer.
   - This affects any workflow where bookies are decommissioned via `bookkeeper shell decommissionbookie` while writers hold open `LedgerHandle` objects whose ensembles include the decommissioned bookies — a routine operational procedure.
   - Reproduced on two independent Pulsar/BookKeeper releases (3.0.x / 4.0.x), suggesting this is a fundamental design gap rather than a version-specific regression.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] addEntry succeeds on replacement bookies after ensemble recovery, but entry is silently dropped — lh.close() fails with BKMetadataVersionException [bookkeeper]

["Shawyeok (via GitHub)" <gi...@apache.org>]

Shawyeok commented on issue #4812:
URL: https://github.com/apache/bookkeeper/issues/4812#issuecomment-4598357773

   ## Reproduction scripts
   
   A self-contained Docker-based reproduction is available as a Gist:
   https://gist.github.com/Shawyeok/a7fc47ee358575b607799db192cf5dde
   
   Files:
   - **`scenario3-pulsar4.sh`** — orchestration script (bootstraps ZK + 3 bookies, runs the client, decommissions and replaces bookies, signals the client to write entry-1)
   - **`LedgerScenario.java`** — the BookKeeper client (opens ledger E=2/WQ=2/AQ=2, writes entry-0, waits for `/tmp/proceed`, then writes entry-1 and closes in separate try-catch blocks)
   
   **Prerequisites:** Docker with `apachepulsar/pulsar:4.0.10` image available.
   
   **Run:**
   ```bash
   bash scenario3-pulsar4.sh
   ```
   
   The final summary lines show the result:
   ```
   addEntry result : [client] entry-1 id=1  LAC=1
   lh.close result : [client] ERROR lh.close(): BKMetadataVersionException: Bad ledger metadata version
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org