[I] Enhancement: Add message-count сompaction policy as alternatives to backlog-size threshold [pulsar]

["Ksnz (via GitHub)" <gi...@apache.org>]

Ksnz opened a new issue, #26091:
URL: https://github.com/apache/pulsar/issues/26091

   ### Search before reporting
   
   - [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
   
   
   ### Motivation
   
   In topics where messages are roughly uniform in size, a byte-based threshold works as a reasonable approximation: bytes accumulated since last compaction correlates well enough with the number of new key updates.
   This assumption breaks down when message sizes are heterogeneous. Consider a topic that mixes small control events (a few hundred bytes) with large snapshot messages (megabytes):
   
   A low threshold is hit after just a handful of large snapshots, triggering compaction runs that are disproportionate to the actual number of stale keys — wasting broker I/O and ledger resources
   A high threshold is never reached during stretches of small-message activity, even when thousands of key updates have accumulated and the compacted view is significantly out of date
   
   The core mismatch is that bytes ≠ number of key updates. Compaction's purpose is to retain only the latest value per key; the amount of work it needs to do — and how stale the compacted view is — is determined by how many messages have been written since the last run, not by their total byte size.
   A message-count threshold addresses this directly: compaction fires after N new messages regardless of their individual sizes, providing a stable and predictable trigger that reflects actual compaction pressure rather than storage volume.
   
   ### Solution
   
   _No response_
   
   ### Alternatives
   
   Tuning topicCompactionThresholdInBytes per topic — already possible, but operationally expensive on large clusters with heterogeneous topics and does not solve the fundamental mismatch between bytes and message staleness
   External compaction triggering via Admin API — works as a workaround but requires additional infrastructure and loses the simplicity of built-in automation
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Enhancement: Add message-count сompaction policy as alternatives to backlog-size threshold [pulsar]

["merlimat (via GitHub)" <gi...@apache.org>]

merlimat closed issue #26091: Enhancement: Add message-count сompaction policy as alternatives to backlog-size threshold
URL: https://github.com/apache/pulsar/issues/26091


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Enhancement: Add message-count сompaction policy as alternatives to backlog-size threshold [pulsar]

["merlimat (via GitHub)" <gi...@apache.org>]

merlimat commented on issue #26091:
URL: https://github.com/apache/pulsar/issues/26091#issuecomment-6388660496

   Thanks @Ksnz. Automatic compaction is still triggered only by the byte threshold (`brokerServiceCompactionThresholdInBytes` and the namespace/topic `compactionThreshold` policy). A message-count trigger would add a broker setting, a new namespace/topic policy, and admin and CLI APIs. It also has to define how it combines with the byte threshold (either one triggers, or one replaces the other). That needs a PIP: https://github.com/apache/pulsar/blob/master/pip/README.md. Closing for now; a PIP is welcome.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org