[GitHub] [pulsar] aloyszhang opened a new issue #9038: Introduce continuous offset for pulsar

[GitBox <gi...@apache.org>]

aloyszhang opened a new issue #9038:
URL: https://github.com/apache/pulsar/issues/9038


   **Is your feature request related to a problem? Please describe.**
   Currently, pulsar provides <LedgerId, EntryId> to locate an Entry, and provides `BatchIndex` for the inner message of Entry.
   But there is not a  continuous offset for each Message in Entry and we could not use the offset to search an Entry. 
   
   For protocol handler like KOP, the map between kakfa offset and pulsar position is implenment by split 64 bits to three parts which represents ledgerId, entryId and batchIndex seperatelly. This way makes the max range of ledgerId much lower than Long.MAX_VALUE and ledgerId space will used up quickly, see https://github.com/streamnative/kop/issues/175 .
   
   **Describe the solution you'd like**
   We should introduce  continuous offset for pulsar, which can be achieved by broker entry metadata introduced in PIP-70.
   
   **Additional context**
   We can introduce offset for each message first, and with this feature, we can 
   1. implement a more stable offset manager for KOP 
   2. provide more preceise metrics on message level like backlog
   3. ...
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui closed issue #9038: Introduce continuous offset for pulsar

[GitBox <gi...@apache.org>]

codelipenghui closed issue #9038:
URL: https://github.com/apache/pulsar/issues/9038


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org