[GitHub] [pulsar] Demogorgon314 opened a new pull request, #21156: [branch-3.0][fix][broker] Avoid splitting one batch message into two entries in StrategicTwoPhaseCompactor

["Demogorgon314 (via GitHub)" <gi...@apache.org>]

Demogorgon314 opened a new pull request, #21156:
URL: https://github.com/apache/pulsar/pull/21156

   (cherry picked from commit e59c850753f2a1d08e4df63bf8862c7cb5db4e71)
   
   ### Motivation
   
   Currently, the `ServiceUnitStateChannelImpl` will send batch messages to `loadbalancer-service-unit-state` topic,
   and the `StrategicTwoPhaseCompactor` will consume the message and rebatch the message to a new batch.
   
   However, the `StrategicTwoPhaseCompactor` may write messages from the same batch into different entries, which will result in skipping some messages when reading compacted entries.
   
   For example, we have a batch: `[(3:0:-1:0, k1), (3:0:-1:1, k2)]`, after compact, the batch maybe `[3:0:-1:0, k1]` and `[3:0:-1:0, k2]` , then if we only read one entry, the `readPosition` will be updated to `3:1` , so the `[3:0:-1:0, k2]` will be skipped.
   
   ### Modifications
   
   * Check the add message `entryId` and `ledgerId` before adding a message to `RawBatchMessageContainerImpl`.
   * Remove size limit for `RawBatchMessageContainerImpl` to let input batch decide.
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Technoboy- merged pull request #21156: [fix][broker] Avoid splitting one batch message into two entries in StrategicTwoPhaseCompactor

["Technoboy- (via GitHub)" <gi...@apache.org>]

Technoboy- merged PR #21156:
URL: https://github.com/apache/pulsar/pull/21156


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org