You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "AuroraTwinkle (via GitHub)" <gi...@apache.org> on 2025/06/03 10:00:10 UTC

Re: [PR] [feat][client] Support Pull Consumer For Pulsar [pulsar-java-contrib]

AuroraTwinkle commented on code in PR #13:
URL: https://github.com/apache/pulsar-java-contrib/pull/13#discussion_r2123325582


##########
pcip/pcip-4.md:
##########
@@ -0,0 +1,179 @@
+# PCIP-4: Pull Consumer Implementation for Apache Pulsar
+
+# Background knowledge
+
+- **Pulsar Consumers**: Pulsar currently supports push-based consumption models (exclusive/shared/failover/key-shared).
+  This proposal adds pull-based consumption.
+- **Message Positioning**: Pulsar uses composite MessageIDs (ledgerId + entryId + partitionId), while systems like
+  Kafka/RocketMQ use monotonic offsets.
+- **Offset Mapping**: https://github.com/apache/pulsar/pull/24220 can be used to convert between offsets and Pulsar's
+  MessageIDs.
+
+# Motivation
+
+System Migration Requirement: The organization plans to migrate from RocketMQ to Pulsar, requiring a unified MQ client
+abstraction layer to conceal implementation details and support seamless engine replacement.
+
+Interface Compatibility Issues:
+
+- Pulsar lacks a native offset retrieval interface (pull/fetch model).
+- RocketMQ/Kafka use monotonically increasing numeric offsets to locate messages, whereas Pulsar employs a composite
+  MessageID (ledgerId + entryId + partitionId).
+
+Objective: Implement a RocketMQ-like Pull Consumer to support precise offset control and reduce migration costs.
+
+# Goals
+
+## In Scope
+
+| Goal                       | Description                                                               |  
+|----------------------------|---------------------------------------------------------------------------|  
+| **Precise Offset Control** | Supports specifying partition, offset, pull count, and byte size.         |  
+| **Resource Efficiency**    | Reuses Reader connections with LRU cache management.                      |  
+| **Easy Integration**       | Compatible with Pulsar’s existing API, requiring no Broker modifications. |  
+
+## Out of Scope
+
+NA
+
+# High Level Design
+
+```mermaid
+graph TD
+    A[PullConsumer] -->|pull| B[Offset Mapping]
+    B -->|convert| C[MessageID]
+    C -->|seek| D[Reader Pool]
+    D -->|fetch| E[Broker]
+    A -->|ack| F[Offset-Based Ack]
+    F -->|convert| G[MessageID]
+    G -->|cumulative ack| H[Consumer]
+```
+
+Key components:
+
+1. **`PulsarPullConsumer` interface**: Standard pull consumer API
+2. **Offset ↔ MessageID Cache**: Partition-scoped mapping layer
+3. **Reader Pool**: Managed resource pool with LRU eviction
+4. **Partition Locking**: Thread-safe access coordination
+
+## Detailed Design
+
+### Design & Implementation Details
+
+**Core Interface** `PulsarPullConsumer`:
+
+```java
+public interface PulsarPullConsumer<T> extends AutoCloseable {
+    void start() throws PulsarClientException;
+
+    List<Message<T>> pull(PullRequest request);
+
+    void ack(long offset, int partition) throws PulsarClientException;
+
+    long searchOffset(int partition, long timestamp) throws PulsarAdminException;

Review Comment:
   Maybe use `index` instead of `offset`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org