[I] [Bug] Asynchronous KV flush can publish a high watermark inside a WAL batch [fluss]

["platinumhamburg (via GitHub)" <gi...@apache.org>]

platinumhamburg opened a new issue, #4267:
URL: https://github.com/apache/fluss/issues/4267

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   
   ### Fluss version
   
   0.9.0 (latest release)
   
   ### Please describe the bug 🐞
   
   The asynchronous KV flush prepares a WAL prefix and splits its KV entries into native RocksDB writes according to entry-count and byte budgets. After each successful segment, it publishes that segment's `flushedLogOffset`. If a later segment encounters storage backpressure, the completed prefix remains applied and the remaining segments are retried later.
   
   The segment boundary is currently the next KV entry's LSN, which need not be a WAL batch boundary. The leader can use this partial flush offset as its high watermark. WAL reads bounded by that high watermark cannot expose the containing batch, even though the logical high watermark advertises progress inside it.
   
   #### Reproduction
   
   1. Append a single WAL batch containing 600 KV inserts, followed by another batch.
   2. Run the scheduled KV flush with the normal 500-entry native write budget.
   3. Allow the first native write to reach real RocksDB, then inject `StorageBackpressureException` immediately before the second native write.
   4. Observe `flushedLogOffset = 500`, inside the first WAL batch `[0, 600)`.
   
   The same test with a 64-byte write budget publishes offset 4 inside that batch. These are deterministic assertion failures against the unmodified segmentation logic; the test does not assign a fabricated flush offset.
   
   #### Expected behavior
   
   Every independently completed asynchronous flush prefix should end at a WAL batch boundary. If storage backpressure interrupts a flush, RocksDB data, row count, and completed flush progress should describe a prefix consisting of whole WAL batches. The leader should not receive a new partial-batch offset to publish as HW.
   
   #### Impact
   
   Ordinary primary-key tables can exhibit an advertised HW with no complete readable batch at the reader's current offset. Consumers that interpret this as a WAL gap can fail instead of waiting. HW is also used when truncating replicas during role changes, making batch alignment relevant to recovery correctness. This report does not claim proven permanent data loss.
   
   ### Solution
   
   Track pending WAL batch end offsets alongside the KV pre-write buffer, register them after successful non-duplicate appends, and reconstruct them during WAL recovery. Split prepared flushes only between complete batches, with each independently completed segment written atomically. Preserve existing asynchronous scheduling, backpressure retries, locks, and HW publication callbacks. A single WAL batch may exceed the native write grouping budget. No persisted format change is required.
   
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Asynchronous KV flush can publish a high watermark inside a WAL batch [fluss]

["platinumhamburg (via GitHub)" <gi...@apache.org>]

platinumhamburg closed issue #4267: [Bug] Asynchronous KV flush can publish a high watermark inside a WAL batch
URL: https://github.com/apache/fluss/issues/4267


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org