You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "grishaf (via GitHub)" <gi...@apache.org> on 2026/05/19 08:17:30 UTC

[PR] [fix][broker] Fix non-batched null-value messages not removed during topic compaction [pulsar]

grishaf opened a new pull request, #25817:
URL: https://github.com/apache/pulsar/pull/25817

   ### Motivation
   When a non-Java producer (C++, Python, Go) sends a null-value message (tombstone) on a compacted topic, the key is not removed during topic compaction. The tombstone is retained in the compacted view instead of being deleted.
   **Root cause:**
   `AbstractTwoPhaseCompactor.extractKeyAndSize()` computed payload size using `headersAndPayload.readableBytes()`, which returns the combined size of the serialized MessageMetadata + payload. For null-value messages the payload is empty, but the metadata is always present, so `readableBytes()` is always > 0 (e.g. 32 bytes of metadata). This prevents the tombstone path (`size <= 0 → latestForKey.remove(key)`) from being reached in phase one of compaction.
   The batch code path is not affected because it extracts per-message payload sizes from `SingleMessageMetadata.payloadSize`, which correctly returns 0 for null-value messages. The Java client always sets `numMessagesInBatch` in the metadata (even with `enableBatching(false)`), so all Java-produced messages go through the batch path — which is why this bug was never caught by existing tests.
   ### Modifications
   - **`AbstractTwoPhaseCompactor.extractKeyAndSize()`**: Fixed to compute the correct payload-only size by using `Commands.skipMessageMetadata()` to skip past the metadata before reading `readableBytes()`. For compressed messages, `getUncompressedSize()` was already correct.
   - **`EventTimeOrderCompactor.extractMessageCompactionData()`**: Refactored to reuse `extractKeyAndSize()` instead of duplicating the (buggy) size calculation.
   ### Verifying this change
   This change added tests and can be verified as follows:
   - Added `CompactionTest.testNonBatchedMessageWithNullValue` — writes raw non-batch entries (no `numMessagesInBatch` in metadata, simulating C++/Python producers) directly to the managed ledger, triggers compaction, and verifies tombstoned keys are removed from the compacted view.
   | Test                                | Without fix                        | With fix |
   |-------------------------------------|------------------------------------|----------|
   | testNonBatchedMessageWithNullValue  | FAILED: expected [2] but found [4] | PASSED   |
   ### Does this pull request potentially affect one of the following parts:
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][broker] Fix non-batched null-value messages not removed during topic compaction [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #25817:
URL: https://github.com/apache/pulsar/pull/25817#issuecomment-4532960541

   /pulsarbot rerun


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][broker] Fix non-batched null-value messages not removed during topic compaction [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #25817:
URL: https://github.com/apache/pulsar/pull/25817


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][broker] Fix non-batched null-value messages not removed during topic compaction [pulsar]

Posted by "grishaf (via GitHub)" <gi...@apache.org>.
grishaf commented on PR #25817:
URL: https://github.com/apache/pulsar/pull/25817#issuecomment-4488339496

   PR failed on a known flaky test
   https://github.com/apache/pulsar/issues/25815


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][broker] Fix non-batched null-value messages not removed during topic compaction [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #25817:
URL: https://github.com/apache/pulsar/pull/25817#issuecomment-4533029808

   I had to trigger a full rebuild since "Pulsar CI" currently caches build artifacts for 3 days and if the artifacts expire, there's a need to trigger a full rebuild. Merging origin/master to the PR branch is a recommended approach. Another way to trigger is to push a commit or close and re-open the PR (however that's not a recommended approach for retrying after flaky tests, `/pulsarbot rerun` should be used instead unless artifacts have expired).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][broker] Fix non-batched null-value messages not removed during topic compaction [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #25817:
URL: https://github.com/apache/pulsar/pull/25817#issuecomment-4532968331

   @grishaf Please avoid force pushes to a PR where the review has already started since it makes it harder to track changes after the previous review. Instead, merge the latest `origin/master` to your PR branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][broker] Fix non-batched null-value messages not removed during topic compaction [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #25817:
URL: https://github.com/apache/pulsar/pull/25817#discussion_r3265915097


##########
pulsar-broker/src/main/java/org/apache/pulsar/compaction/AbstractTwoPhaseCompactor.java:
##########
@@ -462,14 +462,22 @@ private CompletableFuture<Void> addToCompactedLedger(LedgerHandle lh, RawMessage
     return bkf;
   }
 
+  /**
+   * Extract the partition key and the payload size for a non-batch message.
+   *
+   * @return a pair of (partitionKey, payloadSize), or null if the message has no partition key.
+   */
   protected Pair<String, Integer> extractKeyAndSize(RawMessage m, MessageMetadata msgMetadata) {
-    ByteBuf headersAndPayload = m.getHeadersAndPayload();
     if (msgMetadata.hasPartitionKey()) {
-      int size = headersAndPayload.readableBytes();
+      int payloadSize;
       if (msgMetadata.hasUncompressedSize()) {
-        size = msgMetadata.getUncompressedSize();
+        payloadSize = msgMetadata.getUncompressedSize();
+      } else {
+        ByteBuf headersAndPayload = m.getHeadersAndPayload();
+        Commands.skipMessageMetadata(headersAndPayload);
+        payloadSize = headersAndPayload.readableBytes();

Review Comment:
   It's better to call `ByteBuf.duplicate()` so that the original headerAndPayload buffer's readIndex doesn't advance:
   ```suggestion
           ByteBuf headersAndPayload = m.getHeadersAndPayload().duplicate();
           Commands.skipMessageMetadata(headersAndPayload);
           payloadSize = headersAndPayload.readableBytes();
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][broker] Fix non-batched null-value messages not removed during topic compaction [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #25817:
URL: https://github.com/apache/pulsar/pull/25817#issuecomment-4532959244

   > PR failed on a known flaky test #25815
   
   @grishaf btw. in case of flaky tests, you can trigger a retry for the CI build by adding a comment `/pulsarbot rerun` to the PR. This works only after the CI workflow jobs have all finished.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][broker] Fix non-batched null-value messages not removed during topic compaction [pulsar]

Posted by "Denovo1998 (via GitHub)" <gi...@apache.org>.
Denovo1998 commented on code in PR #25817:
URL: https://github.com/apache/pulsar/pull/25817#discussion_r3303886313


##########
pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java:
##########
@@ -640,6 +640,80 @@ public void testBatchMessageWithNullValue() throws Exception {
         assertEquals(messages.get(2).getKey(), "key5");
     }
 
+    /**
+     * Write raw non-batch entries directly to the managed ledger, simulating
+     * messages from C++/Python clients that do not set numMessagesInBatch.
+     * Verifies that null-value tombstones remove keys during compaction.

Review Comment:
   Small wording nit: this comment says the raw entries simulate C++/Python clients because they do not set `numMessagesInBatch`. The Java non-batch send path also does not set `MessageMetadata.numMessagesInBatch`; Java avoids the old bug mainly because `ProducerImpl#updateMessageMetadata` sets `uncompressedSize=0` for null payloads.
   
   Could we reword this as "raw non-batch entries without `uncompressedSize`, as seen with some non-Java clients" so future readers don't infer that `numMessagesInBatch` is what protects Java non-batch messages?



##########
pulsar-broker/src/main/java/org/apache/pulsar/compaction/AbstractTwoPhaseCompactor.java:
##########
@@ -462,14 +462,22 @@ private CompletableFuture<Void> addToCompactedLedger(LedgerHandle lh, RawMessage
     return bkf;
   }
 
+  /**
+   * Extract the partition key and the payload size for a non-batch message.
+   *
+   * @return a pair of (partitionKey, payloadSize), or null if the message has no partition key.
+   */
   protected Pair<String, Integer> extractKeyAndSize(RawMessage m, MessageMetadata msgMetadata) {
-    ByteBuf headersAndPayload = m.getHeadersAndPayload();
     if (msgMetadata.hasPartitionKey()) {
-      int size = headersAndPayload.readableBytes();
+      int payloadSize;
       if (msgMetadata.hasUncompressedSize()) {
-        size = msgMetadata.getUncompressedSize();
+        payloadSize = msgMetadata.getUncompressedSize();
+      } else {
+        ByteBuf headersAndPayload = m.getHeadersAndPayload().duplicate();
+        Commands.skipMessageMetadata(headersAndPayload);
+        payloadSize = headersAndPayload.readableBytes();

Review Comment:
   Should we check `msgMetadata.hasNullValue() && msgMetadata.isNullValue()` before deriving the size from payload bytes?
   
   The new payload-only calculation fixes plain non-batch tombstones, but it can still keep encrypted tombstones when the producer does not set `uncompressedSize`. Encryption can turn an empty/null payload into non-empty ciphertext, so after `skipMessageMetadata(...)`, `readableBytes()` may still be > 0 and phase one will put the tombstone into `latestForKey` instead of removing the key.
   
   Java producers usually avoid this because `ProducerImpl#updateMessageMetadata` sets `uncompressedSize=0` for null payloads, but this PR is specifically fixing producers whose metadata does not include that size. The explicit `nullValue` flag is the protocol-level tombstone signal, so I think this helper should return size 0 for `nullValue=true` before falling back to `uncompressedSize` or payload bytes.
   
   ```java
   if (msgMetadata.hasNullValue() && msgMetadata.isNullValue()) {
       payloadSize = 0;
   } else if (msgMetadata.hasUncompressedSize()) {
       payloadSize = msgMetadata.getUncompressedSize();
   } else {
       ByteBuf headersAndPayload = m.getHeadersAndPayload().duplicate();
       Commands.skipMessageMetadata(headersAndPayload);
       payloadSize = headersAndPayload.readableBytes();
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org