[PR] [fix][txn] PIP-473: prune terminal transactions from the metadata buffer cache [pulsar]

["merlimat (via GitHub)" <gi...@apache.org>]

merlimat opened a new pull request, #25960:
URL: https://github.com/apache/pulsar/pull/25960

   ### Motivation
   
   `MetadataTransactionBuffer` kept an in-memory entry for **every** transaction it ever saw (committed and aborted alike) in its `txns` cache, and never removed them — so the map grew unbounded for the segment's lifetime. On long-lived segments with high transaction throughput this is a memory leak.
   
   ### Modifications
   
   - `applyTerminalNow`: drop the entry from `txns` once the transaction is terminal, bounding the cache to the current open-transaction count. This is safe:
     - `recomputeMaxReadPositionLocked` only consults `OPEN` entries, so dropping terminal ones doesn't affect the max-read position;
     - `isTxnAborted` reads the **separate** `abortedTxns` set — an aborted txn stays filtered there, and a committed/unknown txn correctly reads as visible;
     - the positions needed for the durable side-effects (aborted record, watermark, op-record cleanup) are captured before the entry is removed.
   - Add `trackedTxnCount()` (`@VisibleForTesting`) and a unit test asserting the cache is pruned back to empty after a mix of commits and aborts, while visibility stays correct (aborted filtered, committed visible).
   
   `MetadataPendingAckStore.openTxns` is already pruned on terminal, and `/txn` headers are already GC'd by the coordinator's GC sweep, so this change targets the one remaining unbounded structure.
   
   ### Out of scope (follow-up)
   
   The durable per-segment aborted records and the `abortedTxns` set grow only with **aborts** and are meant to be pruned on segment-ledger trim. That trim-driven GC needs a ledger-trim trigger that doesn't exist yet (`TransactionBuffer.purgeTxns` is declared but never invoked anywhere — the legacy buffer no-ops it too), so it's left as a follow-up.
   
   ### Verifying this change
   
   - `MetadataTransactionBufferTest` — 13/13 (12 existing + the new `terminalTxns_prunedFromCache_visibilityUnchanged`).
   - `V5TransactionTest` end-to-end commit/abort — passing.
   - checkstyle clean.
   
   ### Does this pull request potentially affect one of the following parts:
   
   - Dependencies: no
   - The public API: no
   - The schema: no
   - The default values of configurations: no
   - The threading model: no
   - The binary protocol: no
   - The REST endpoints: no
   - The admin CLI options: no
   - Anything that affects deployment: no
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [fix][txn] PIP-473: prune terminal transactions from the metadata buffer cache [pulsar]

["merlimat (via GitHub)" <gi...@apache.org>]

merlimat merged PR #25960:
URL: https://github.com/apache/pulsar/pull/25960


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org