You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "liangyepianzhou (via GitHub)" <gi...@apache.org> on 2025/05/16 14:43:38 UTC
Re: [PR] [feat][pip]PIP-415: Support getting message ID by Index [pulsar]
liangyepianzhou commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2092573473
##########
pip/pip-415.md:
##########
@@ -0,0 +1,152 @@
+# PIP-415: Support getting message ID by index
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the index of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+**We can retrieve the message index using a message ID, but lack the reverse capability to obtain message IDs by index. This breaks the logical circularity and creates operational confusion.**
+Then we need to add a new API to get the message id by index
+# Motivation
+
+Our organization is currently planning a migration from RocketMQ to Pulsar. To facilitate this transition, we aim to implement a standardized abstraction layer for MQ clients that encapsulates implementation details of specific messaging systems. This abstraction layer will allow seamless engine replacement while maintaining consistent client interfaces.
+However, one critical compatibility issues hinder the unification of message fetching pat terns between RocketMQ/Kafka and Pulsar:
+
+**Positioning Mechanism Mismatch**:
+- RocketMQ/Kafka: Utilize monotonically increasing numerical index for message positioning and acknowledgment
+- Pulsar: Relies on composite MessageID (ledgerId + entryId + batchIndex) for message identification
+
+We propose to add a new API to retrieve the message ID by index, enabling us to cache the mapping between message ID and index.
+This will allow us to use index for seek and acknowledgment operations when consuming messages through the standardized API.
+
+# Goals
+
+## In Scope
+
+- Add a new API to retrieve the message ID by index
+
+# High Level Design
+
+# Detailed Design
+The core implementation involves adding a new HTTP endpoint to query MessageId by index. Underlying implementation will leverage existing `Broker Entry Metadata` containing index values.
+
+## Design & Implementation Details
+
+## Public-facing Changes
+
+### Public API
+
+Add a new admin API endpoint `getMessageIdByIndex` to `org.apache.pulsar.broker.admin.v2.PersistentTopics`.
+#### Request Path
+- **Path Template**:
+ ```http
+ GET /{tenant}/{namespace}/{topic}/getMessageIdByIndex
+ ```
+- `{tenant}`: Tenant name (string, path parameter).
+- `{namespace}`: Namespace name (string, path parameter).
+- `{topic}`: Topic name (URL-encoded string, path parameter).
+
+#### HTTP Method
+- **Method**: `GET`
+- **Purpose**: Retrieve the message ID associated with a specific message index in a topic.
+
+#### Parameters
+
+| Type | Name | Data Type | Required | Description |
+|-----------------|-----------------|-----------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Path Param** | `tenant` | `String` | Yes | Tenant identifier for multi-tenancy isolation. |
+| | `namespace` | `String` | Yes | Namespace identifier to group topics under a tenant. |
+| | `topic` | `String` | Yes | Topic Name - either the specific partition name of a partitioned topic (e.g. my-topic-partition-0) or the original topic name for non-partitioned topics. |
+| **Query Param** | `index` | `long` | Yes | Zero-based index of the message in the topic. |
+| | `authoritative` | `boolean` | No | If authoritative is true, it means the lookup had already been redirected here by a different broker. Default: `false`. |
+
+#### Response
+
+##### Success Response
+- **Status Code**: `200 OK`
+- **Body Format**:
+ ```json
+ {
+ "ledgerId": 12345,
+ "entryId": 67890,
+ "partitionIndex": 0
+ }
+ ```
+ - `ledgerId`: ID of the ledger where the message is stored.
+ - `entryId`: Unique identifier of the message within the ledger.
+ - `partitionIndex`: Partition index (relevant for partitioned topics; `-1` for non-partitioned topics).
Review Comment:
Thank you for your suggestions. I agree that users should be explicitly alerted to these details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org