You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "davidthorne72 (via GitHub)" <gi...@apache.org> on 2026/10/21 14:21:18 UTC
[I] Bulk Acknowledge can skip producer-batched message IDs; batch tracking causes severe ACK lock contention [pulsar-dotpulsar]
davidthorne72 opened a new issue, #298:
URL: https://github.com/apache/pulsar-dotpulsar/issues/298
### Description
IConsumer<T>.Acknowledge(IEnumerable<MessageId>) can complete without processing all supplied IDs when the messages came from Pulsar producer batches.
SubConsumer puts every supplied ID into one CommandAck, but ConsumerChannel.Send(CommandAck) passes only command.MessageId[0] through BatchHandler.Acknowledge. If that first ID does not complete its producer batch, Send returns without inspecting the remaining IDs or sending an ACK command. If it does complete a batch, the remaining IDs are sent without being processed through batch tracking.
There is also a performance problem as outstanding producer batches accumulate. BatchHandler.Acknowledge searches a linked list under a shared lock for every batched ID. Removing a completed batch by value searches the list again, and checking whether a batch is complete scans its bit array. Message receipt and batch addition use the same lock. Thus incomplete tracked batches can make ACK calls progressively slower and block other consumer activity.
### Reproduction Steps
1. Use DotPulsar 5.3.2 with a producer that puts at least two messages in one Pulsar producer batch. A producer supporting batching can be used to generate the input; DotPulsar itself need only be the consumer.
2. Receive both messages on a fresh subscription. Verify their MessageIds have the same ledger ID, entry ID, and partition, but different nonnegative BatchIndex values.
3. Acknowledge both in one call, with the first batch index first:
var first = await consumer.Receive(ct);
var second = await consumer.Receive(ct);
// Preconditions: first and second belong to the same producer batch.
await consumer.Acknowledge(
new[] { first.MessageId, second.MessageId }, ct);
### Expected behavior
Every ID supplied to the bulk call is processed. Batched message indices are tracked individually, and an ACK for their enclosing producer batch is sent once all its indices have been acknowledged. Independent unbatched IDs are sent regardless of whether another producer batch is still incomplete.
### Actual behavior
ConsumerChannel.Send examines only the first ID for batch tracking. When that ID does not complete its batch, it returns without processing the remaining IDs or sending a broker ACK. The public bulk-ACK call can nevertheless complete normally. Outstanding messages remain on the subscription, while incomplete batch entries remain in the client’s tracker.
### Regression?
Unknown. The behavior was investigated with DotPulsar 5.3.2. Source inspection found the relevant paths unchanged in 5.3.4; this report does not claim a runtime reproduction on 5.3.4.
### Known Workarounds
Calling Acknowledge(MessageId) separately for each ID avoids the bulk method’s skipped-ID path. It does not fix the linked-list search and monitor contention, so it is a correctness workaround rather than an established high-throughput solution.
### Configuration
- DotPulsar: 5.3.2
- Application target: .NET 10
- Workload: producer-batched messages (batch size of 1000), bulk individual acknowledgments, and a high-volume consumer.
- OS/distro and architecture: Ubuntu 2204, GCE N2 128-highmem machine on consumer side, n2-highmem-64 on producer side
### Other information
The source paths inspected at the commit identified by the 5.3.2 NuGet package are:
- SubConsumer.InternalAcknowledge(IEnumerable<MessageId>) (https://github.com/apache/pulsar-dotpulsar/blob/6fa4f2666f6b32d366d777930152d9574a3cf38c/src/DotPulsar/Internal/SubConsumer.cs): constructs the multi-ID ACK command.
- ConsumerChannel.Send(CommandAck) (https://github.com/apache/pulsar-dotpulsar/blob/6fa4f2666f6b32d366d777930152d9574a3cf38c/src/DotPulsar/Internal/ConsumerChannel.cs): processes only the first ID through batch tracking and may return before sending.
- BatchHandler.Acknowledge (https://github.com/apache/pulsar-dotpulsar/blob/6fa4f2666f6b32d366d777930152d9574a3cf38c/src/DotPulsar/Internal/BatchHandler.cs): searches and removes tracked batches under a shared lock.
A .nettrace captured during a high-volume producer-batched workload provides additional performance evidence. dotnet-trace report topN attributed 78.43% exclusive call-stack time to Monitor.Enter_Slowpath; DotPulsar’s BatchHandler.Acknowledge(MessageIdData) appeared on that ACK stack with 78.41% inclusive time. This indicates substantial ACK-path monitor contention. These are call-stack-time figures, not a claim that 78% of CPU cycles execute ACK instructions. The trace supports the performance problem; the bulk skipped-ID behavior follows from source inspection and should be covered by the reproduction above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org