You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "jcmfernandes (via GitHub)" <gi...@apache.org> on 2026/04/20 16:42:56 UTC
[PR] [pulsar-client-go/`Reader`] fix: deliver null-value tombstones instead of discarding them [pulsar-client-go]
jcmfernandes opened a new pull request, #1482:
URL: https://github.com/apache/pulsar-client-go/pull/1482
<!--
### Contribution Checklist
- Name the pull request in the form "[Issue XYZ][component] Title of the pull request", where *XYZ* should be replaced by the actual issue number.
Skip *Issue XYZ* if there is no associated github issue for this pull request.
Skip *component* if you are unsure about which is the best component. E.g. `[docs] Fix typo in produce method`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
### Motivation
A message published with `MessageMetadata.null_value = true` (the Pulsar compaction tombstone convention) was conflated with a deserialization failure in `partitionConsumer.MessageReceived` and silently discarded. Because `lastDequeuedMsg` was never advanced past the tombstone, `hasMoreMessages` kept returning true and `Reader.Next` blocked forever when a tombstone was the last message on a topic.
### Modifications
Consumer: when the reader yields an empty payload and message metadata or the single-message metadata has `null_value` set, build a normal message with `payLoad == nil` and take the usual dispatch path so `lastDequeuedMsg` advances. Real corruption still routes through `discardCorruptedMessage`.
Producer: set `MessageMetadata.null_value`/`SingleMessageMetadata.null_value` when both `Value` and `Payload` are nil, matching the Java client so Go-produced tombstones carry the flag consumers need.
Message gains an `IsNullValue()` bool accessor so applications can tell tombstones apart from empty payloads.
**Note:** This PR was written with the assistance of AI Anthropic’s Opus 4.7.
### Verifying this change
This change added tests and can be verified as follows:
- Added reader test
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (~~yes~~ / **no**)
- The public API: (**yes** / ~~no~~) (added `IsNullValue()` to the `Message` interface)
- The schema: (~~yes~~ / **no** / ~~don't know~~)
- The default values of configurations: (~~yes~~ / **no**)
- The wire protocol: (~~yes~~ / **no**)
### Documentation
- Does this pull request introduce a new feature? (~~yes~~ / **no**)
- If yes, how is the feature documented? (not applicable / docs / GoDocs / not documented): **N/A.**
- If a feature is not applicable for documentation, explain why? **N/A.**
- If a feature is not documented yet in this PR, please create a followup issue for adding the documentation. **N/A.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Reader] fix: deliver null-value tombstones instead of discarding them [pulsar-client-go]
Posted by "jcmfernandes (via GitHub)" <gi...@apache.org>.
jcmfernandes commented on PR #1482:
URL: https://github.com/apache/pulsar-client-go/pull/1482#issuecomment-4334137835
Thank you very much, @nodece! And thanks for helping me get this merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Reader] fix: deliver null-value tombstones instead of discarding them [pulsar-client-go]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on code in PR #1482:
URL: https://github.com/apache/pulsar-client-go/pull/1482#discussion_r3121544638
##########
pulsar/consumer_partition.go:
##########
@@ -1308,10 +1308,25 @@ func (pc *partitionConsumer) MessageReceived(response *pb.CommandMessage, header
)
for i := 0; i < numMsgs; i++ {
smm, payload, err := reader.ReadMessage()
- if err != nil || payload == nil {
+ isNullValue := msgMeta.GetNullValue() || (smm != nil && smm.GetNullValue())
+ if err != nil {
+ // A null-value (tombstone) message has no payload bytes on the wire, so
+ // the non-batched reader returns ErrEOM. Accept it instead of discarding
+ // it as corrupted, matching the Java client's behavior for compaction
+ // tombstones.
+ if isNullValue && err == internal.ErrEOM {
+ payload = nil
+ } else {
+ pc.discardCorruptedMessage(pbMsgID, pb.CommandAck_BatchDeSerializeError)
+ return err
+ }
Review Comment:
I know that this logic looks hacky, `ReadMessage` should be improved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Reader] fix: deliver null-value tombstones instead of discarding them [pulsar-client-go]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on PR #1482:
URL: https://github.com/apache/pulsar-client-go/pull/1482#issuecomment-4331772466
Good work @jcmfernandes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Reader] fix: deliver null-value tombstones instead of discarding them [pulsar-client-go]
Posted by "jcmfernandes (via GitHub)" <gi...@apache.org>.
jcmfernandes commented on PR #1482:
URL: https://github.com/apache/pulsar-client-go/pull/1482#issuecomment-4329131871
> @jcmfernandes Please fix lint.
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Reader] fix: deliver null-value tombstones instead of discarding them [pulsar-client-go]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece merged PR #1482:
URL: https://github.com/apache/pulsar-client-go/pull/1482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Reader] fix: deliver null-value tombstones instead of discarding them [pulsar-client-go]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on PR #1482:
URL: https://github.com/apache/pulsar-client-go/pull/1482#issuecomment-4325050992
@jcmfernandes Please fix lint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Reader] fix: deliver null-value tombstones instead of discarding them [pulsar-client-go]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1482:
URL: https://github.com/apache/pulsar-client-go/pull/1482#discussion_r3120394700
##########
pulsar/consumer_partition.go:
##########
@@ -1308,10 +1308,25 @@ func (pc *partitionConsumer) MessageReceived(response *pb.CommandMessage, header
)
for i := 0; i < numMsgs; i++ {
smm, payload, err := reader.ReadMessage()
- if err != nil || payload == nil {
+ isNullValue := msgMeta.GetNullValue() || (smm != nil && smm.GetNullValue())
+ if err != nil {
+ // A null-value (tombstone) message has no payload bytes on the wire, so
+ // the non-batched reader returns ErrEOM. Accept it instead of discarding
+ // it as corrupted, matching the Java client's behavior for compaction
+ // tombstones.
+ if isNullValue && err == internal.ErrEOM {
+ payload = nil
+ } else {
+ pc.discardCorruptedMessage(pbMsgID, pb.CommandAck_BatchDeSerializeError)
+ return err
+ }
Review Comment:
When accepting a tombstone with `err == internal.ErrEOM`, the code keeps `err` non-nil and continues. It works today because `err` is not used later, but it’s easy to misread and could become a latent bug if future logic inspects/returns `err` after this block. Consider explicitly setting `err = nil` in the tombstone/`ErrEOM` acceptance path to make the intent unambiguous.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Reader] fix: deliver null-value tombstones instead of discarding them [pulsar-client-go]
Posted by "jcmfernandes (via GitHub)" <gi...@apache.org>.
jcmfernandes commented on code in PR #1482:
URL: https://github.com/apache/pulsar-client-go/pull/1482#discussion_r3122599981
##########
pulsar/consumer_partition.go:
##########
@@ -1308,10 +1308,25 @@ func (pc *partitionConsumer) MessageReceived(response *pb.CommandMessage, header
)
for i := 0; i < numMsgs; i++ {
smm, payload, err := reader.ReadMessage()
- if err != nil || payload == nil {
+ isNullValue := msgMeta.GetNullValue() || (smm != nil && smm.GetNullValue())
+ if err != nil {
+ // A null-value (tombstone) message has no payload bytes on the wire, so
+ // the non-batched reader returns ErrEOM. Accept it instead of discarding
+ // it as corrupted, matching the Java client's behavior for compaction
+ // tombstones.
+ if isNullValue && err == internal.ErrEOM {
+ payload = nil
+ } else {
+ pc.discardCorruptedMessage(pbMsgID, pb.CommandAck_BatchDeSerializeError)
+ return err
+ }
Review Comment:
It's indeed a bit messy. I addressed Copilot's feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org