You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "liangyepianzhou (via GitHub)" <gi...@apache.org> on 2025/04/27 02:45:22 UTC
[PR] PIP-415: Support getting message ID by offset [pulsar]
liangyepianzhou opened a new pull request, #24220:
URL: https://github.com/apache/pulsar/pull/24220
### Motivation
we can now obtain the offset of a message by its message id:
1. Get the message by id using `get-message-by-id` cmd
2. Get the index of the message using `Message.getIndex()`
But we cannot obtain the message id by offset. Then we need to add a new API to get the message id by offset.
### Modifications
Add a new http API to retrieve the message ID by offset.
We propose to add a new API to retrieve the message ID by offset, enabling us to cache the mapping between message ID and offset.
This will allow us to use offsets for seek and acknowledgment operations when consuming messages through the standardized API.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: <!-- ENTER URL HERE -->
<!--
After opening this PR, the build in apache/pulsar will fail and instructions will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed in
a forked repository.
The tests will be run in the forked repository until all PR review comments have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by offset [pulsar]
Posted by "StevenLuMT (via GitHub)" <gi...@apache.org>.
StevenLuMT commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2062793294
##########
pip/pip-415.md:
##########
@@ -0,0 +1,112 @@
+# PIP-415: Support getting message ID by offset
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the offset of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+But we cannot obtain the message id by offset. Then we need to add a new API to get the message id by offset.
+
+# Motivation
+
Review Comment:
I think we should not blindly compare pulsar and Rocketmq. This is not convincing at all. I think we can look at it from our own perspective.
Motivation is written relatively simply. I feel that some index planning can be added. Currently, the index function is open ( brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor
), but some functions are actually imperfect, such as:
Get MessageId through index
Get the corresponding index through MessageId
This time, the function of getting MessageId through index will be improved first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by Index [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2094839540
##########
pip/pip-415.md:
##########
@@ -0,0 +1,186 @@
+# PIP-415: Support getting message ID by index
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the index of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+**We can retrieve the message index using a message ID, but lack the reverse capability to obtain message IDs by index. This breaks the logical circularity and creates operational confusion.**
+Then we need to add a new API to get the message id by index
+# Motivation
+
+Our organization is currently planning a migration from RocketMQ to Pulsar. To facilitate this transition, we aim to implement a standardized abstraction layer for MQ clients that encapsulates implementation details of specific messaging systems. This abstraction layer will allow seamless engine replacement while maintaining consistent client interfaces.
+However, one critical compatibility issues hinder the unification of message fetching pat terns between RocketMQ/Kafka and Pulsar:
+
+**Positioning Mechanism Mismatch**:
+- RocketMQ/Kafka: Utilize monotonically increasing numerical index for message positioning and acknowledgment
+- Pulsar: Relies on composite MessageID (ledgerId + entryId + batchIndex) for message identification
+
+We propose to add a new API to retrieve the message ID by index, enabling us to cache the mapping between message ID and index.
+This will allow us to use index for seek and acknowledgment operations when consuming messages through the standardized API.
+
+# Goals
+
+## In Scope
+
+- Add a new API to retrieve the message ID by index
+
+# High Level Design
+
+# Detailed Design
+The core implementation involves adding a new HTTP endpoint to query MessageId by index. Underlying implementation will leverage existing `Broker Entry Metadata` containing index values.
+
+## Design & Implementation Details
+
+## Public-facing Changes
+
+### Public API
+
+Add a new admin API endpoint `getMessageIdByIndex` to `org.apache.pulsar.broker.admin.v2.PersistentTopics`.
+#### Request Path
+- **Path Template**:
+ ```http
+ GET /{tenant}/{namespace}/{topic}/getMessageIdByIndex
+ ```
+- `{tenant}`: Tenant name (string, path parameter).
+- `{namespace}`: Namespace name (string, path parameter).
+- `{topic}`: Topic name (URL-encoded string, path parameter).
+
+#### HTTP Method
+- **Method**: `GET`
+- **Purpose**: Retrieve the message ID associated with a specific message index in a topic.
+
+#### Parameters
+
+| Type | Name | Data Type | Required | Description |
+|-----------------|-----------------|-----------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Path Param** | `tenant` | `String` | Yes | Tenant identifier for multi-tenancy isolation. |
+| | `namespace` | `String` | Yes | Namespace identifier to group topics under a tenant. |
+| | `topic` | `String` | Yes | Topic Name - either the specific partition name of a partitioned topic (e.g. my-topic-partition-0) or the original topic name for non-partitioned topics. |
+| **Query Param** | `index` | `long` | Yes | Zero-based index of the message in the topic. |
+| | `authoritative` | `boolean` | No | If authoritative is true, it means the lookup had already been redirected here by a different broker. Default: `false`. |
+
+#### Response
+
+##### Success Response
+- **Status Code**: `200 OK`
+- **Body Format**:
+ ```json
+ {
+ "ledgerId": 12345,
+ "entryId": 67890,
+ "partitionIndex": 0
+ }
+ ```
+ - `ledgerId`: ID of the ledger where the message is stored.
+ - `entryId`: Unique identifier of the message within the ledger.
+ - `partitionIndex`: Partition index (relevant for partitioned topics; `-1` for non-partitioned topics).
+
+**notes: If the specified index points to a system message, return the first valid user message following this index.**
+
+When retrieving a message ID by index, the resolution is limited to the **entry** level (an entry is the minimal storage unit for messages in Pulsar's persistence layer). If message batching is enabled, a single entry may contain multiple messages with distinct indexes.
+
+**Example Scenario** (partition with 2 entries):
+
+| Entry | Ledger ID | Entry ID | Index | Messages |
+| :--- | ---: | ---: | ---: | ---: |
+| A | 0 | 0 | 2 | 0,1,2 |
+| B | 0 | 1 | 4 | 3,4 |
+
+- Param indexes 0,1,2 or 3,4 will return the **same MessageID** (e.g., `MessageId(0:0:*)` for Entry A).
+- To achieve **precise message-level consumption**, you can first locate the entry using the MessageID, then filter messages by their indexes during client-side processing.
+ - Example: To consume from index 1, retrieve all messages in Entry A and filter out messages with indexes below 1.
+
+**Why Precise Index Matching Isn't Implemented on the Broker Side:**
+1. **Metadata Parsing Constraints**: Precise matching requires parsing `MessageMetadata`, but brokers are designed to parse only `BrokerEntryMetadata`.
+2. **Limited Benefits**: Even if the exact `batchIndex` and `batchSize` are determined, the broker must still send the **entire entry** to the client, offering no network bandwidth savings.
+3. **Client-Side Efficiency**: Once the client receives the entry, users can directly filter messages by index (after parsing `MessageMetadata`, which includes unique indexes for each message).
+
+##### Error Responses
+
+| Status Code | Description |
+|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **307 Temporary Redirect** | The current broker does not own this topic. Redirect to the correct broker. |
+| **403 Forbidden** | Client lacks permissions to access the topic/namespace. |
+| **404 Not Found** | Topic/namespace does not exist, or the specified `index` is invalid. An index value of -1 or one that exceeds the maximum index range is considered invalid. |
Review Comment:
The `MessageId` is mainly used for `seek` as the proposal documents
> We propose to add a new API to retrieve the message ID by index, enabling us to cache the mapping between message ID and index.
> This will allow us to use index for seek and acknowledgment operations when consuming messages through the standardized API.
So the cache might be:
- t0: 0 => (0, 0), 1 => (0, 1), 2 => (1, 0)
- t1: 0 => (0, 1), 1 => (0, 1), 2 => (1, 0)
- t2: 0 => (1, 0), 1 => (1, 0), 2 => (1, 0)
Returning the first valid message's position cannot differ if the message of the index has been removed.
If you care much about the seek case, returning `MessageId.earliest` could achieve the same goal, where it can also tell users the message of the given index has been removed.
- t0: 0 => (0, 0), 1 => (0, 1), 2 => (1, 0)
- t1: 0 => earliest, 1 => (0, 1), 2 => (1, 0)
- t2: 0 => earliest, 1 => earliest, 2 => (1, 0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by offset [pulsar]
Posted by "StevenLuMT (via GitHub)" <gi...@apache.org>.
StevenLuMT commented on PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#issuecomment-2833779509
Motivation is written relatively simply. I feel that some index planning can be added. Currently, the index function is open ( brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor
), but some functions are actually imperfect, such as:
1. Get MessageId through index
2. Get the corresponding index through MessageId
This time, the function of getting MessageId through index will be improved first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by offset [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2062799901
##########
pip/pip-415.md:
##########
@@ -0,0 +1,112 @@
+# PIP-415: Support getting message ID by offset
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the offset of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+But we cannot obtain the message id by offset. Then we need to add a new API to get the message id by offset.
+
+# Motivation
+
+Our organization is currently planning a migration from RocketMQ to Pulsar. To facilitate this transition, we aim to implement a standardized abstraction layer for MQ clients that encapsulates implementation details of specific messaging systems. This abstraction layer will allow seamless engine replacement while maintaining consistent client interfaces.
+However, one critical compatibility issues hinder the unification of message fetching pat terns between RocketMQ/Kafka and Pulsar:
+
+**Positioning Mechanism Mismatch**:
+- RocketMQ/Kafka: Utilize monotonically increasing numerical offsets for message positioning and acknowledgment
+- Pulsar: Relies on composite MessageID (ledgerId + entryId + batchIndex) for message identification
+
+We propose to add a new API to retrieve the message ID by offset, enabling us to cache the mapping between message ID and offset.
+This will allow us to use offsets for seek and acknowledgment operations when consuming messages through the standardized API.
+
+# Goals
+
+## In Scope
+
+- Add a new API to retrieve the message ID by offset
+
+# High Level Design
+
+# Detailed Design
+The core implementation involves adding a new HTTP endpoint to query MessageId by offset (index). Underlying implementation will leverage existing `Broker Entry Metadata` containing index values.
+
+## Design & Implementation Details
+
+## Public-facing Changes
+
+### Public API
+
+```java
+ @GET
+ @Path("/{tenant}/{namespace}/{topic}/getMessageIdByOffset")
+ @ApiOperation(hidden = true, value = "Get Message ID by offset.",
+ notes = "If the specified offset is a system message, "
+ + "it will return the message id of the later message.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 307, message = "Current broker doesn't serve the namespace of this topic"),
+ @ApiResponse(code = 403, message = "Don't have admin permission"),
+ @ApiResponse(code = 404, message = "Namespace or partitioned topic does not exist, "
+ + "or the offset is invalid")})
+ public void getMessageIDByOffsetAndPartitionID(@Suspended final AsyncResponse asyncResponse,
+ @PathParam("tenant") String tenant,
+ @PathParam("namespace") String namespace,
+ @PathParam("topic") @Encoded String encodedTopic,
+ @QueryParam("offset") long offset,
+ @QueryParam("authoritative") @DefaultValue("false")
+ boolean authoritative){
+ validateTopicName(tenant, namespace, encodedTopic);
+ internalGetMessageIDByOffsetAsync(offset, authoritative)
+ .thenAccept(asyncResponse::resume)
+ .exceptionally(ex -> {
+ if (!isRedirectException(ex)) {
+ log.error("[{}] Failed to get message id by offset for topic {}, partition id {}, offset {}",
+ clientAppId(), topicName, offset, ex);
+ }
+ resumeAsyncResponseExceptionally(asyncResponse, ex);
+ return null;
+ });
+ }
+```
+
+### Binary protocol
+
+### Configuration
+
+### CLI
+Add new subcommand to pulsar-admin:
+
+```bash
+ pulsar-admin topics get-message-id-by-offset \
Review Comment:
Since we're using "index" as the concept, it's better to replace "offset" with "index" in all commands and methods.
##########
pip/pip-415.md:
##########
@@ -0,0 +1,112 @@
+# PIP-415: Support getting message ID by offset
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the offset of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+But we cannot obtain the message id by offset. Then we need to add a new API to get the message id by offset.
+
+# Motivation
+
+Our organization is currently planning a migration from RocketMQ to Pulsar. To facilitate this transition, we aim to implement a standardized abstraction layer for MQ clients that encapsulates implementation details of specific messaging systems. This abstraction layer will allow seamless engine replacement while maintaining consistent client interfaces.
+However, one critical compatibility issues hinder the unification of message fetching pat terns between RocketMQ/Kafka and Pulsar:
+
+**Positioning Mechanism Mismatch**:
+- RocketMQ/Kafka: Utilize monotonically increasing numerical offsets for message positioning and acknowledgment
+- Pulsar: Relies on composite MessageID (ledgerId + entryId + batchIndex) for message identification
+
+We propose to add a new API to retrieve the message ID by offset, enabling us to cache the mapping between message ID and offset.
+This will allow us to use offsets for seek and acknowledgment operations when consuming messages through the standardized API.
+
+# Goals
+
+## In Scope
+
+- Add a new API to retrieve the message ID by offset
+
+# High Level Design
+
+# Detailed Design
+The core implementation involves adding a new HTTP endpoint to query MessageId by offset (index). Underlying implementation will leverage existing `Broker Entry Metadata` containing index values.
+
+## Design & Implementation Details
+
+## Public-facing Changes
+
+### Public API
+
+```java
+ @GET
+ @Path("/{tenant}/{namespace}/{topic}/getMessageIdByOffset")
+ @ApiOperation(hidden = true, value = "Get Message ID by offset.",
+ notes = "If the specified offset is a system message, "
+ + "it will return the message id of the later message.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 307, message = "Current broker doesn't serve the namespace of this topic"),
+ @ApiResponse(code = 403, message = "Don't have admin permission"),
+ @ApiResponse(code = 404, message = "Namespace or partitioned topic does not exist, "
+ + "or the offset is invalid")})
+ public void getMessageIDByOffsetAndPartitionID(@Suspended final AsyncResponse asyncResponse,
+ @PathParam("tenant") String tenant,
+ @PathParam("namespace") String namespace,
+ @PathParam("topic") @Encoded String encodedTopic,
+ @QueryParam("offset") long offset,
+ @QueryParam("authoritative") @DefaultValue("false")
+ boolean authoritative){
+ validateTopicName(tenant, namespace, encodedTopic);
+ internalGetMessageIDByOffsetAsync(offset, authoritative)
+ .thenAccept(asyncResponse::resume)
+ .exceptionally(ex -> {
+ if (!isRedirectException(ex)) {
+ log.error("[{}] Failed to get message id by offset for topic {}, partition id {}, offset {}",
+ clientAppId(), topicName, offset, ex);
+ }
+ resumeAsyncResponseExceptionally(asyncResponse, ex);
+ return null;
+ });
Review Comment:
```suggestion
```
No need to mention any implementation here.
In addition, you only mention the HTTP protocol here. It seems that you should add the client side API (`PulsarAdmin`) as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by Index [pulsar]
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2094914660
##########
pip/pip-415.md:
##########
@@ -0,0 +1,186 @@
+# PIP-415: Support getting message ID by index
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the index of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+**We can retrieve the message index using a message ID, but lack the reverse capability to obtain message IDs by index. This breaks the logical circularity and creates operational confusion.**
+Then we need to add a new API to get the message id by index
+# Motivation
+
+Our organization is currently planning a migration from RocketMQ to Pulsar. To facilitate this transition, we aim to implement a standardized abstraction layer for MQ clients that encapsulates implementation details of specific messaging systems. This abstraction layer will allow seamless engine replacement while maintaining consistent client interfaces.
+However, one critical compatibility issues hinder the unification of message fetching pat terns between RocketMQ/Kafka and Pulsar:
+
+**Positioning Mechanism Mismatch**:
+- RocketMQ/Kafka: Utilize monotonically increasing numerical index for message positioning and acknowledgment
+- Pulsar: Relies on composite MessageID (ledgerId + entryId + batchIndex) for message identification
+
+We propose to add a new API to retrieve the message ID by index, enabling us to cache the mapping between message ID and index.
+This will allow us to use index for seek and acknowledgment operations when consuming messages through the standardized API.
+
+# Goals
+
+## In Scope
+
+- Add a new API to retrieve the message ID by index
+
+# High Level Design
+
+# Detailed Design
+The core implementation involves adding a new HTTP endpoint to query MessageId by index. Underlying implementation will leverage existing `Broker Entry Metadata` containing index values.
+
+## Design & Implementation Details
+
+## Public-facing Changes
+
+### Public API
+
+Add a new admin API endpoint `getMessageIdByIndex` to `org.apache.pulsar.broker.admin.v2.PersistentTopics`.
+#### Request Path
+- **Path Template**:
+ ```http
+ GET /{tenant}/{namespace}/{topic}/getMessageIdByIndex
+ ```
+- `{tenant}`: Tenant name (string, path parameter).
+- `{namespace}`: Namespace name (string, path parameter).
+- `{topic}`: Topic name (URL-encoded string, path parameter).
+
+#### HTTP Method
+- **Method**: `GET`
+- **Purpose**: Retrieve the message ID associated with a specific message index in a topic.
+
+#### Parameters
+
+| Type | Name | Data Type | Required | Description |
+|-----------------|-----------------|-----------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Path Param** | `tenant` | `String` | Yes | Tenant identifier for multi-tenancy isolation. |
+| | `namespace` | `String` | Yes | Namespace identifier to group topics under a tenant. |
+| | `topic` | `String` | Yes | Topic Name - either the specific partition name of a partitioned topic (e.g. my-topic-partition-0) or the original topic name for non-partitioned topics. |
+| **Query Param** | `index` | `long` | Yes | Zero-based index of the message in the topic. |
+| | `authoritative` | `boolean` | No | If authoritative is true, it means the lookup had already been redirected here by a different broker. Default: `false`. |
+
+#### Response
+
+##### Success Response
+- **Status Code**: `200 OK`
+- **Body Format**:
+ ```json
+ {
+ "ledgerId": 12345,
+ "entryId": 67890,
+ "partitionIndex": 0
+ }
+ ```
+ - `ledgerId`: ID of the ledger where the message is stored.
+ - `entryId`: Unique identifier of the message within the ledger.
+ - `partitionIndex`: Partition index (relevant for partitioned topics; `-1` for non-partitioned topics).
+
+**notes: If the specified index points to a system message, return the first valid user message following this index.**
+
+When retrieving a message ID by index, the resolution is limited to the **entry** level (an entry is the minimal storage unit for messages in Pulsar's persistence layer). If message batching is enabled, a single entry may contain multiple messages with distinct indexes.
+
+**Example Scenario** (partition with 2 entries):
+
+| Entry | Ledger ID | Entry ID | Index | Messages |
+| :--- | ---: | ---: | ---: | ---: |
+| A | 0 | 0 | 2 | 0,1,2 |
+| B | 0 | 1 | 4 | 3,4 |
+
+- Param indexes 0,1,2 or 3,4 will return the **same MessageID** (e.g., `MessageId(0:0:*)` for Entry A).
+- To achieve **precise message-level consumption**, you can first locate the entry using the MessageID, then filter messages by their indexes during client-side processing.
+ - Example: To consume from index 1, retrieve all messages in Entry A and filter out messages with indexes below 1.
+
+**Why Precise Index Matching Isn't Implemented on the Broker Side:**
+1. **Metadata Parsing Constraints**: Precise matching requires parsing `MessageMetadata`, but brokers are designed to parse only `BrokerEntryMetadata`.
+2. **Limited Benefits**: Even if the exact `batchIndex` and `batchSize` are determined, the broker must still send the **entire entry** to the client, offering no network bandwidth savings.
+3. **Client-Side Efficiency**: Once the client receives the entry, users can directly filter messages by index (after parsing `MessageMetadata`, which includes unique indexes for each message).
+
+##### Error Responses
+
+| Status Code | Description |
+|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **307 Temporary Redirect** | The current broker does not own this topic. Redirect to the correct broker. |
+| **403 Forbidden** | Client lacks permissions to access the topic/namespace. |
+| **404 Not Found** | Topic/namespace does not exist, or the specified `index` is invalid. An index value of -1 or one that exceeds the maximum index range is considered invalid. |
Review Comment:
Good suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by Index [pulsar]
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2093167268
##########
pip/pip-415.md:
##########
@@ -0,0 +1,186 @@
+# PIP-415: Support getting message ID by index
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the index of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+**We can retrieve the message index using a message ID, but lack the reverse capability to obtain message IDs by index. This breaks the logical circularity and creates operational confusion.**
+Then we need to add a new API to get the message id by index
+# Motivation
+
+Our organization is currently planning a migration from RocketMQ to Pulsar. To facilitate this transition, we aim to implement a standardized abstraction layer for MQ clients that encapsulates implementation details of specific messaging systems. This abstraction layer will allow seamless engine replacement while maintaining consistent client interfaces.
+However, one critical compatibility issues hinder the unification of message fetching pat terns between RocketMQ/Kafka and Pulsar:
+
+**Positioning Mechanism Mismatch**:
+- RocketMQ/Kafka: Utilize monotonically increasing numerical index for message positioning and acknowledgment
+- Pulsar: Relies on composite MessageID (ledgerId + entryId + batchIndex) for message identification
+
+We propose to add a new API to retrieve the message ID by index, enabling us to cache the mapping between message ID and index.
+This will allow us to use index for seek and acknowledgment operations when consuming messages through the standardized API.
+
+# Goals
+
+## In Scope
+
+- Add a new API to retrieve the message ID by index
+
+# High Level Design
+
+# Detailed Design
+The core implementation involves adding a new HTTP endpoint to query MessageId by index. Underlying implementation will leverage existing `Broker Entry Metadata` containing index values.
+
+## Design & Implementation Details
+
+## Public-facing Changes
+
+### Public API
+
+Add a new admin API endpoint `getMessageIdByIndex` to `org.apache.pulsar.broker.admin.v2.PersistentTopics`.
+#### Request Path
+- **Path Template**:
+ ```http
+ GET /{tenant}/{namespace}/{topic}/getMessageIdByIndex
+ ```
+- `{tenant}`: Tenant name (string, path parameter).
+- `{namespace}`: Namespace name (string, path parameter).
+- `{topic}`: Topic name (URL-encoded string, path parameter).
+
+#### HTTP Method
+- **Method**: `GET`
+- **Purpose**: Retrieve the message ID associated with a specific message index in a topic.
+
+#### Parameters
+
+| Type | Name | Data Type | Required | Description |
+|-----------------|-----------------|-----------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Path Param** | `tenant` | `String` | Yes | Tenant identifier for multi-tenancy isolation. |
+| | `namespace` | `String` | Yes | Namespace identifier to group topics under a tenant. |
+| | `topic` | `String` | Yes | Topic Name - either the specific partition name of a partitioned topic (e.g. my-topic-partition-0) or the original topic name for non-partitioned topics. |
+| **Query Param** | `index` | `long` | Yes | Zero-based index of the message in the topic. |
+| | `authoritative` | `boolean` | No | If authoritative is true, it means the lookup had already been redirected here by a different broker. Default: `false`. |
+
+#### Response
+
+##### Success Response
+- **Status Code**: `200 OK`
+- **Body Format**:
+ ```json
+ {
+ "ledgerId": 12345,
+ "entryId": 67890,
+ "partitionIndex": 0
+ }
+ ```
+ - `ledgerId`: ID of the ledger where the message is stored.
+ - `entryId`: Unique identifier of the message within the ledger.
+ - `partitionIndex`: Partition index (relevant for partitioned topics; `-1` for non-partitioned topics).
+
+**notes: If the specified index points to a system message, return the first valid user message following this index.**
+
+When retrieving a message ID by index, the resolution is limited to the **entry** level (an entry is the minimal storage unit for messages in Pulsar's persistence layer). If message batching is enabled, a single entry may contain multiple messages with distinct indexes.
+
+**Example Scenario** (partition with 2 entries):
+
+| Entry | Ledger ID | Entry ID | Index | Messages |
+| :--- | ---: | ---: | ---: | ---: |
+| A | 0 | 0 | 2 | 0,1,2 |
+| B | 0 | 1 | 4 | 3,4 |
+
+- Param indexes 0,1,2 or 3,4 will return the **same MessageID** (e.g., `MessageId(0:0:*)` for Entry A).
+- To achieve **precise message-level consumption**, you can first locate the entry using the MessageID, then filter messages by their indexes during client-side processing.
+ - Example: To consume from index 1, retrieve all messages in Entry A and filter out messages with indexes below 1.
+
+**Why Precise Index Matching Isn't Implemented on the Broker Side:**
+1. **Metadata Parsing Constraints**: Precise matching requires parsing `MessageMetadata`, but brokers are designed to parse only `BrokerEntryMetadata`.
+2. **Limited Benefits**: Even if the exact `batchIndex` and `batchSize` are determined, the broker must still send the **entire entry** to the client, offering no network bandwidth savings.
+3. **Client-Side Efficiency**: Once the client receives the entry, users can directly filter messages by index (after parsing `MessageMetadata`, which includes unique indexes for each message).
+
+##### Error Responses
+
+| Status Code | Description |
+|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **307 Temporary Redirect** | The current broker does not own this topic. Redirect to the correct broker. |
+| **403 Forbidden** | Client lacks permissions to access the topic/namespace. |
+| **404 Not Found** | Topic/namespace does not exist, or the specified `index` is invalid. An index value of -1 or one that exceeds the maximum index range is considered invalid. |
Review Comment:
> It's not correct. For example, if old messages are trimmed (e.g. message 0,1,2), the left messages might be 4,5, then the index where index < 4 will be invalid.
I hold a different view. If old messages are trimmed, the system should return the first valid message. Similarly, if an index points to a system marker, it should return the first valid message after that marker.
> In addition, this REST API should only work when BrokerEntryMetadataInterceptor is configured and exposingBrokerEntryMetadataToClientEnabled is true, right? IMO, in this case, we should return a 500 Internal Server Error
Correct. The server must enable `BrokerEntryMetadataInterceptor`, but for this specific API, `exposingBrokerEntryMetadataToClientEnabled` might not be strictly mandatory. Although, in practice, these two configurations are often required together.
For such cases, I opted to return **412 Precondition Failed**. I deliberated extensively between 412 and 500 but ultimately concluded that 412 is more appropriate. This is not a server-side error but rather a failure to meet request prerequisites—the request was sent to a broker cluster lacking the required configuration. Thus, it qualifies as a precondition error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by offset [pulsar]
Posted by "StevenLuMT (via GitHub)" <gi...@apache.org>.
StevenLuMT commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2062793294
##########
pip/pip-415.md:
##########
@@ -0,0 +1,112 @@
+# PIP-415: Support getting message ID by offset
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the offset of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+But we cannot obtain the message id by offset. Then we need to add a new API to get the message id by offset.
+
+# Motivation
+
Review Comment:
I think we should not blindly compare Pulsar and RocketMQ. This is not convincing at all. I think we can look at it from pulsar own perspective.
Motivation is written relatively simply. I feel that some index planning can be added. Currently, the index function is open ( brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor
), but some functions are actually imperfect, such as:
Get MessageId through index
Get the corresponding index through MessageId
This time, the function of getting MessageId through index will be improved first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by offset [pulsar]
Posted by "StevenLuMT (via GitHub)" <gi...@apache.org>.
StevenLuMT commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2062793294
##########
pip/pip-415.md:
##########
@@ -0,0 +1,112 @@
+# PIP-415: Support getting message ID by offset
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the offset of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+But we cannot obtain the message id by offset. Then we need to add a new API to get the message id by offset.
+
+# Motivation
+
Review Comment:
I think we should not blindly compare pulsar and Rocketmq. This is not convincing at all. I think we can look at it from pulsar own perspective.
Motivation is written relatively simply. I feel that some index planning can be added. Currently, the index function is open ( brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor
), but some functions are actually imperfect, such as:
Get MessageId through index
Get the corresponding index through MessageId
This time, the function of getting MessageId through index will be improved first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by offset [pulsar]
Posted by "StevenLuMT (via GitHub)" <gi...@apache.org>.
StevenLuMT commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2062793294
##########
pip/pip-415.md:
##########
@@ -0,0 +1,112 @@
+# PIP-415: Support getting message ID by offset
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the offset of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+But we cannot obtain the message id by offset. Then we need to add a new API to get the message id by offset.
+
+# Motivation
+
Review Comment:
I think we should not blindly compare Pulsar and RocketMQ. This is not convincing at all. I think we can look at it from Pulsar own perspective.
Motivation is written relatively simply. I feel that some index planning can be added. Currently, the index function is open ( brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor
), but some functions are actually imperfect, such as:
Get MessageId through index
Get the corresponding index through MessageId
This time, the function of getting MessageId through index will be improved first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by Index [pulsar]
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2092573473
##########
pip/pip-415.md:
##########
@@ -0,0 +1,152 @@
+# PIP-415: Support getting message ID by index
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the index of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+**We can retrieve the message index using a message ID, but lack the reverse capability to obtain message IDs by index. This breaks the logical circularity and creates operational confusion.**
+Then we need to add a new API to get the message id by index
+# Motivation
+
+Our organization is currently planning a migration from RocketMQ to Pulsar. To facilitate this transition, we aim to implement a standardized abstraction layer for MQ clients that encapsulates implementation details of specific messaging systems. This abstraction layer will allow seamless engine replacement while maintaining consistent client interfaces.
+However, one critical compatibility issues hinder the unification of message fetching pat terns between RocketMQ/Kafka and Pulsar:
+
+**Positioning Mechanism Mismatch**:
+- RocketMQ/Kafka: Utilize monotonically increasing numerical index for message positioning and acknowledgment
+- Pulsar: Relies on composite MessageID (ledgerId + entryId + batchIndex) for message identification
+
+We propose to add a new API to retrieve the message ID by index, enabling us to cache the mapping between message ID and index.
+This will allow us to use index for seek and acknowledgment operations when consuming messages through the standardized API.
+
+# Goals
+
+## In Scope
+
+- Add a new API to retrieve the message ID by index
+
+# High Level Design
+
+# Detailed Design
+The core implementation involves adding a new HTTP endpoint to query MessageId by index. Underlying implementation will leverage existing `Broker Entry Metadata` containing index values.
+
+## Design & Implementation Details
+
+## Public-facing Changes
+
+### Public API
+
+Add a new admin API endpoint `getMessageIdByIndex` to `org.apache.pulsar.broker.admin.v2.PersistentTopics`.
+#### Request Path
+- **Path Template**:
+ ```http
+ GET /{tenant}/{namespace}/{topic}/getMessageIdByIndex
+ ```
+- `{tenant}`: Tenant name (string, path parameter).
+- `{namespace}`: Namespace name (string, path parameter).
+- `{topic}`: Topic name (URL-encoded string, path parameter).
+
+#### HTTP Method
+- **Method**: `GET`
+- **Purpose**: Retrieve the message ID associated with a specific message index in a topic.
+
+#### Parameters
+
+| Type | Name | Data Type | Required | Description |
+|-----------------|-----------------|-----------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Path Param** | `tenant` | `String` | Yes | Tenant identifier for multi-tenancy isolation. |
+| | `namespace` | `String` | Yes | Namespace identifier to group topics under a tenant. |
+| | `topic` | `String` | Yes | Topic Name - either the specific partition name of a partitioned topic (e.g. my-topic-partition-0) or the original topic name for non-partitioned topics. |
+| **Query Param** | `index` | `long` | Yes | Zero-based index of the message in the topic. |
+| | `authoritative` | `boolean` | No | If authoritative is true, it means the lookup had already been redirected here by a different broker. Default: `false`. |
+
+#### Response
+
+##### Success Response
+- **Status Code**: `200 OK`
+- **Body Format**:
+ ```json
+ {
+ "ledgerId": 12345,
+ "entryId": 67890,
+ "partitionIndex": 0
+ }
+ ```
+ - `ledgerId`: ID of the ledger where the message is stored.
+ - `entryId`: Unique identifier of the message within the ledger.
+ - `partitionIndex`: Partition index (relevant for partitioned topics; `-1` for non-partitioned topics).
Review Comment:
Thank you for your suggestions. I agree that users should be explicitly alerted to these details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by Index [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2093906165
##########
pip/pip-415.md:
##########
@@ -0,0 +1,186 @@
+# PIP-415: Support getting message ID by index
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the index of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+**We can retrieve the message index using a message ID, but lack the reverse capability to obtain message IDs by index. This breaks the logical circularity and creates operational confusion.**
+Then we need to add a new API to get the message id by index
+# Motivation
+
+Our organization is currently planning a migration from RocketMQ to Pulsar. To facilitate this transition, we aim to implement a standardized abstraction layer for MQ clients that encapsulates implementation details of specific messaging systems. This abstraction layer will allow seamless engine replacement while maintaining consistent client interfaces.
+However, one critical compatibility issues hinder the unification of message fetching pat terns between RocketMQ/Kafka and Pulsar:
+
+**Positioning Mechanism Mismatch**:
+- RocketMQ/Kafka: Utilize monotonically increasing numerical index for message positioning and acknowledgment
+- Pulsar: Relies on composite MessageID (ledgerId + entryId + batchIndex) for message identification
+
+We propose to add a new API to retrieve the message ID by index, enabling us to cache the mapping between message ID and index.
+This will allow us to use index for seek and acknowledgment operations when consuming messages through the standardized API.
+
+# Goals
+
+## In Scope
+
+- Add a new API to retrieve the message ID by index
+
+# High Level Design
+
+# Detailed Design
+The core implementation involves adding a new HTTP endpoint to query MessageId by index. Underlying implementation will leverage existing `Broker Entry Metadata` containing index values.
+
+## Design & Implementation Details
+
+## Public-facing Changes
+
+### Public API
+
+Add a new admin API endpoint `getMessageIdByIndex` to `org.apache.pulsar.broker.admin.v2.PersistentTopics`.
+#### Request Path
+- **Path Template**:
+ ```http
+ GET /{tenant}/{namespace}/{topic}/getMessageIdByIndex
+ ```
+- `{tenant}`: Tenant name (string, path parameter).
+- `{namespace}`: Namespace name (string, path parameter).
+- `{topic}`: Topic name (URL-encoded string, path parameter).
+
+#### HTTP Method
+- **Method**: `GET`
+- **Purpose**: Retrieve the message ID associated with a specific message index in a topic.
+
+#### Parameters
+
+| Type | Name | Data Type | Required | Description |
+|-----------------|-----------------|-----------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Path Param** | `tenant` | `String` | Yes | Tenant identifier for multi-tenancy isolation. |
+| | `namespace` | `String` | Yes | Namespace identifier to group topics under a tenant. |
+| | `topic` | `String` | Yes | Topic Name - either the specific partition name of a partitioned topic (e.g. my-topic-partition-0) or the original topic name for non-partitioned topics. |
+| **Query Param** | `index` | `long` | Yes | Zero-based index of the message in the topic. |
+| | `authoritative` | `boolean` | No | If authoritative is true, it means the lookup had already been redirected here by a different broker. Default: `false`. |
+
+#### Response
+
+##### Success Response
+- **Status Code**: `200 OK`
+- **Body Format**:
+ ```json
+ {
+ "ledgerId": 12345,
+ "entryId": 67890,
+ "partitionIndex": 0
+ }
+ ```
+ - `ledgerId`: ID of the ledger where the message is stored.
+ - `entryId`: Unique identifier of the message within the ledger.
+ - `partitionIndex`: Partition index (relevant for partitioned topics; `-1` for non-partitioned topics).
+
+**notes: If the specified index points to a system message, return the first valid user message following this index.**
+
+When retrieving a message ID by index, the resolution is limited to the **entry** level (an entry is the minimal storage unit for messages in Pulsar's persistence layer). If message batching is enabled, a single entry may contain multiple messages with distinct indexes.
+
+**Example Scenario** (partition with 2 entries):
+
+| Entry | Ledger ID | Entry ID | Index | Messages |
+| :--- | ---: | ---: | ---: | ---: |
+| A | 0 | 0 | 2 | 0,1,2 |
+| B | 0 | 1 | 4 | 3,4 |
+
+- Param indexes 0,1,2 or 3,4 will return the **same MessageID** (e.g., `MessageId(0:0:*)` for Entry A).
+- To achieve **precise message-level consumption**, you can first locate the entry using the MessageID, then filter messages by their indexes during client-side processing.
+ - Example: To consume from index 1, retrieve all messages in Entry A and filter out messages with indexes below 1.
+
+**Why Precise Index Matching Isn't Implemented on the Broker Side:**
+1. **Metadata Parsing Constraints**: Precise matching requires parsing `MessageMetadata`, but brokers are designed to parse only `BrokerEntryMetadata`.
+2. **Limited Benefits**: Even if the exact `batchIndex` and `batchSize` are determined, the broker must still send the **entire entry** to the client, offering no network bandwidth savings.
+3. **Client-Side Efficiency**: Once the client receives the entry, users can directly filter messages by index (after parsing `MessageMetadata`, which includes unique indexes for each message).
+
+##### Error Responses
+
+| Status Code | Description |
+|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **307 Temporary Redirect** | The current broker does not own this topic. Redirect to the correct broker. |
+| **403 Forbidden** | Client lacks permissions to access the topic/namespace. |
+| **404 Not Found** | Topic/namespace does not exist, or the specified `index` is invalid. An index value of -1 or one that exceeds the maximum index range is considered invalid. |
Review Comment:
> If old messages are trimmed, the system should return the first valid message.
Then if there are two `getMessageIdByIndex` calls with the same index argument, these two calls could both succeed but return different results, which could be confusing to users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by Index [pulsar]
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou merged PR #24220:
URL: https://github.com/apache/pulsar/pull/24220
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by Index [pulsar]
Posted by "StevenLuMT (via GitHub)" <gi...@apache.org>.
StevenLuMT commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2099123096
##########
pip/pip-415.md:
##########
@@ -0,0 +1,152 @@
+# PIP-415: Support getting message ID by index
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the index of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+**We can retrieve the message index using a message ID, but lack the reverse capability to obtain message IDs by index. This breaks the logical circularity and creates operational confusion.**
+Then we need to add a new API to get the message id by index
+# Motivation
+
+Our organization is currently planning a migration from RocketMQ to Pulsar. To facilitate this transition, we aim to implement a standardized abstraction layer for MQ clients that encapsulates implementation details of specific messaging systems. This abstraction layer will allow seamless engine replacement while maintaining consistent client interfaces.
+However, one critical compatibility issues hinder the unification of message fetching pat terns between RocketMQ/Kafka and Pulsar:
+
+**Positioning Mechanism Mismatch**:
+- RocketMQ/Kafka: Utilize monotonically increasing numerical index for message positioning and acknowledgment
+- Pulsar: Relies on composite MessageID (ledgerId + entryId + batchIndex) for message identification
+
+We propose to add a new API to retrieve the message ID by index, enabling us to cache the mapping between message ID and index.
+This will allow us to use index for seek and acknowledgment operations when consuming messages through the standardized API.
+
+# Goals
+
+## In Scope
+
+- Add a new API to retrieve the message ID by index
+
+# High Level Design
+
+# Detailed Design
+The core implementation involves adding a new HTTP endpoint to query MessageId by index. Underlying implementation will leverage existing `Broker Entry Metadata` containing index values.
+
+## Design & Implementation Details
+
+## Public-facing Changes
+
+### Public API
+
+Add a new admin API endpoint `getMessageIdByIndex` to `org.apache.pulsar.broker.admin.v2.PersistentTopics`.
+#### Request Path
+- **Path Template**:
+ ```http
+ GET /{tenant}/{namespace}/{topic}/getMessageIdByIndex
+ ```
+- `{tenant}`: Tenant name (string, path parameter).
+- `{namespace}`: Namespace name (string, path parameter).
+- `{topic}`: Topic name (URL-encoded string, path parameter).
+
+#### HTTP Method
+- **Method**: `GET`
+- **Purpose**: Retrieve the message ID associated with a specific message index in a topic.
+
+#### Parameters
+
+| Type | Name | Data Type | Required | Description |
+|-----------------|-----------------|-----------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Path Param** | `tenant` | `String` | Yes | Tenant identifier for multi-tenancy isolation. |
+| | `namespace` | `String` | Yes | Namespace identifier to group topics under a tenant. |
+| | `topic` | `String` | Yes | Topic Name - either the specific partition name of a partitioned topic (e.g. my-topic-partition-0) or the original topic name for non-partitioned topics. |
+| **Query Param** | `index` | `long` | Yes | Zero-based index of the message in the topic. |
+| | `authoritative` | `boolean` | No | If authoritative is true, it means the lookup had already been redirected here by a different broker. Default: `false`. |
+
+#### Response
+
+##### Success Response
+- **Status Code**: `200 OK`
+- **Body Format**:
+ ```json
+ {
+ "ledgerId": 12345,
+ "entryId": 67890,
+ "partitionIndex": 0
+ }
+ ```
+ - `ledgerId`: ID of the ledger where the message is stored.
+ - `entryId`: Unique identifier of the message within the ledger.
+ - `partitionIndex`: Partition index (relevant for partitioned topics; `-1` for non-partitioned topics).
Review Comment:
This situation of batch can indeed be supported, but a batch returns the same offset, and the engine also conforms to the semantics of batch granularity, which is a good suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by Index [pulsar]
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2094795799
##########
pip/pip-415.md:
##########
@@ -0,0 +1,186 @@
+# PIP-415: Support getting message ID by index
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the index of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+**We can retrieve the message index using a message ID, but lack the reverse capability to obtain message IDs by index. This breaks the logical circularity and creates operational confusion.**
+Then we need to add a new API to get the message id by index
+# Motivation
+
+Our organization is currently planning a migration from RocketMQ to Pulsar. To facilitate this transition, we aim to implement a standardized abstraction layer for MQ clients that encapsulates implementation details of specific messaging systems. This abstraction layer will allow seamless engine replacement while maintaining consistent client interfaces.
+However, one critical compatibility issues hinder the unification of message fetching pat terns between RocketMQ/Kafka and Pulsar:
+
+**Positioning Mechanism Mismatch**:
+- RocketMQ/Kafka: Utilize monotonically increasing numerical index for message positioning and acknowledgment
+- Pulsar: Relies on composite MessageID (ledgerId + entryId + batchIndex) for message identification
+
+We propose to add a new API to retrieve the message ID by index, enabling us to cache the mapping between message ID and index.
+This will allow us to use index for seek and acknowledgment operations when consuming messages through the standardized API.
+
+# Goals
+
+## In Scope
+
+- Add a new API to retrieve the message ID by index
+
+# High Level Design
+
+# Detailed Design
+The core implementation involves adding a new HTTP endpoint to query MessageId by index. Underlying implementation will leverage existing `Broker Entry Metadata` containing index values.
+
+## Design & Implementation Details
+
+## Public-facing Changes
+
+### Public API
+
+Add a new admin API endpoint `getMessageIdByIndex` to `org.apache.pulsar.broker.admin.v2.PersistentTopics`.
+#### Request Path
+- **Path Template**:
+ ```http
+ GET /{tenant}/{namespace}/{topic}/getMessageIdByIndex
+ ```
+- `{tenant}`: Tenant name (string, path parameter).
+- `{namespace}`: Namespace name (string, path parameter).
+- `{topic}`: Topic name (URL-encoded string, path parameter).
+
+#### HTTP Method
+- **Method**: `GET`
+- **Purpose**: Retrieve the message ID associated with a specific message index in a topic.
+
+#### Parameters
+
+| Type | Name | Data Type | Required | Description |
+|-----------------|-----------------|-----------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Path Param** | `tenant` | `String` | Yes | Tenant identifier for multi-tenancy isolation. |
+| | `namespace` | `String` | Yes | Namespace identifier to group topics under a tenant. |
+| | `topic` | `String` | Yes | Topic Name - either the specific partition name of a partitioned topic (e.g. my-topic-partition-0) or the original topic name for non-partitioned topics. |
+| **Query Param** | `index` | `long` | Yes | Zero-based index of the message in the topic. |
+| | `authoritative` | `boolean` | No | If authoritative is true, it means the lookup had already been redirected here by a different broker. Default: `false`. |
+
+#### Response
+
+##### Success Response
+- **Status Code**: `200 OK`
+- **Body Format**:
+ ```json
+ {
+ "ledgerId": 12345,
+ "entryId": 67890,
+ "partitionIndex": 0
+ }
+ ```
+ - `ledgerId`: ID of the ledger where the message is stored.
+ - `entryId`: Unique identifier of the message within the ledger.
+ - `partitionIndex`: Partition index (relevant for partitioned topics; `-1` for non-partitioned topics).
+
+**notes: If the specified index points to a system message, return the first valid user message following this index.**
+
+When retrieving a message ID by index, the resolution is limited to the **entry** level (an entry is the minimal storage unit for messages in Pulsar's persistence layer). If message batching is enabled, a single entry may contain multiple messages with distinct indexes.
+
+**Example Scenario** (partition with 2 entries):
+
+| Entry | Ledger ID | Entry ID | Index | Messages |
+| :--- | ---: | ---: | ---: | ---: |
+| A | 0 | 0 | 2 | 0,1,2 |
+| B | 0 | 1 | 4 | 3,4 |
+
+- Param indexes 0,1,2 or 3,4 will return the **same MessageID** (e.g., `MessageId(0:0:*)` for Entry A).
+- To achieve **precise message-level consumption**, you can first locate the entry using the MessageID, then filter messages by their indexes during client-side processing.
+ - Example: To consume from index 1, retrieve all messages in Entry A and filter out messages with indexes below 1.
+
+**Why Precise Index Matching Isn't Implemented on the Broker Side:**
+1. **Metadata Parsing Constraints**: Precise matching requires parsing `MessageMetadata`, but brokers are designed to parse only `BrokerEntryMetadata`.
+2. **Limited Benefits**: Even if the exact `batchIndex` and `batchSize` are determined, the broker must still send the **entire entry** to the client, offering no network bandwidth savings.
+3. **Client-Side Efficiency**: Once the client receives the entry, users can directly filter messages by index (after parsing `MessageMetadata`, which includes unique indexes for each message).
+
+##### Error Responses
+
+| Status Code | Description |
+|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **307 Temporary Redirect** | The current broker does not own this topic. Redirect to the correct broker. |
+| **403 Forbidden** | Client lacks permissions to access the topic/namespace. |
+| **404 Not Found** | Topic/namespace does not exist, or the specified `index` is invalid. An index value of -1 or one that exceeds the maximum index range is considered invalid. |
Review Comment:
>Then if there are two getMessageIdByIndex calls with the same index argument, these two calls could both succeed but return different results, which could be confusing to users.
Let me explain why returning the first valid message for expired/indexed-out data is acceptable in practical scenarios:
**Primary Application Scenarios**
1. **Management Scenarios** - Users query message IDs or messages via frontend interfaces.
2. **Data Consumption Scenarios** - Clients retrieve message IDs by index to locate and consume messages.
**Behavior Analysis**
*For expired data returning the first valid message:*
1. **Management Scenarios**:
- Users should be explicitly notified that the API returns the first valid message corresponding to the requested index.
- When the index points to system markers or expired messages, a discrepancy between the query index and the actual message index may occur. Clear documentation/UI warnings would prevent confusion.
2. **Data Consumption Scenarios**:
- Clients can directly consume from a valid message position.
- If the returned message index differs from the requested index, clients may log warnings/debug messages without disrupting normal operations.
*Alternative approach (error reporting):*
1. **Management Scenarios**:
- Still requires explaining the "first valid message" semantics. Additional error handling complexity arises for expired indices.
2. **Data Consumption Scenarios**:
- API errors for expired messages would force clients to handle exceptions, re-query starting positions, or consume from earliest. Unhandled errors could crash applications.
**Conclusion**
The semantics of this API inherently resolve to *"return the first valid message"* due to the existence of system markers. Returning 404 for expired messages introduces unnecessary complexity without tangible benefits. A documented behavior (auto-forwarding to first valid message) maintains usability for both management and consumption scenarios while avoiding error-handling overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by offset [pulsar]
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2063023755
##########
pip/pip-415.md:
##########
@@ -0,0 +1,112 @@
+# PIP-415: Support getting message ID by offset
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the offset of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+But we cannot obtain the message id by offset. Then we need to add a new API to get the message id by offset.
+
+# Motivation
+
+Our organization is currently planning a migration from RocketMQ to Pulsar. To facilitate this transition, we aim to implement a standardized abstraction layer for MQ clients that encapsulates implementation details of specific messaging systems. This abstraction layer will allow seamless engine replacement while maintaining consistent client interfaces.
+However, one critical compatibility issues hinder the unification of message fetching pat terns between RocketMQ/Kafka and Pulsar:
+
+**Positioning Mechanism Mismatch**:
+- RocketMQ/Kafka: Utilize monotonically increasing numerical offsets for message positioning and acknowledgment
+- Pulsar: Relies on composite MessageID (ledgerId + entryId + batchIndex) for message identification
+
+We propose to add a new API to retrieve the message ID by offset, enabling us to cache the mapping between message ID and offset.
+This will allow us to use offsets for seek and acknowledgment operations when consuming messages through the standardized API.
+
+# Goals
+
+## In Scope
+
+- Add a new API to retrieve the message ID by offset
+
+# High Level Design
+
+# Detailed Design
+The core implementation involves adding a new HTTP endpoint to query MessageId by offset (index). Underlying implementation will leverage existing `Broker Entry Metadata` containing index values.
+
+## Design & Implementation Details
+
+## Public-facing Changes
+
+### Public API
+
+```java
+ @GET
+ @Path("/{tenant}/{namespace}/{topic}/getMessageIdByOffset")
+ @ApiOperation(hidden = true, value = "Get Message ID by offset.",
+ notes = "If the specified offset is a system message, "
+ + "it will return the message id of the later message.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 307, message = "Current broker doesn't serve the namespace of this topic"),
+ @ApiResponse(code = 403, message = "Don't have admin permission"),
+ @ApiResponse(code = 404, message = "Namespace or partitioned topic does not exist, "
+ + "or the offset is invalid")})
+ public void getMessageIDByOffsetAndPartitionID(@Suspended final AsyncResponse asyncResponse,
+ @PathParam("tenant") String tenant,
+ @PathParam("namespace") String namespace,
+ @PathParam("topic") @Encoded String encodedTopic,
+ @QueryParam("offset") long offset,
+ @QueryParam("authoritative") @DefaultValue("false")
+ boolean authoritative){
+ validateTopicName(tenant, namespace, encodedTopic);
+ internalGetMessageIDByOffsetAsync(offset, authoritative)
+ .thenAccept(asyncResponse::resume)
+ .exceptionally(ex -> {
+ if (!isRedirectException(ex)) {
+ log.error("[{}] Failed to get message id by offset for topic {}, partition id {}, offset {}",
+ clientAppId(), topicName, offset, ex);
+ }
+ resumeAsyncResponseExceptionally(asyncResponse, ex);
+ return null;
+ });
Review Comment:
You are right. I have modified it according to your suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip]PIP-415: Support getting message ID by offset [pulsar]
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #24220:
URL: https://github.com/apache/pulsar/pull/24220#discussion_r2063021903
##########
pip/pip-415.md:
##########
@@ -0,0 +1,112 @@
+# PIP-415: Support getting message ID by offset
+
+# Background knowledge
+
+
+- In [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata), we introduced `Broker Entry Metadata` containing an `index` field - a continuous sequence identifier for messages within a topic partition.
+This sequence ID simplifies message sequence management compared to the composite MessageId (ledgerId+entryId+batchIndex) which becomes discontinuous across ledgers.
+- In [PIP-90](https://github.com/apache/pulsar/wiki/PIP-90%3A-Expose-broker-entry-metadata-to-the-client), we exposed this metadata via the `Message.getIndex()` API.
+- In [PR-6331](https://github.com/apache/pulsar/pull/6331), we added a new `get-message-by-id` cmd into pulsar-admin.
+
+After these three works, we can now obtain the offset of a message by its message id:
+
+1. Get the message by id using `get-message-by-id` cmd
+2. Get the index of the message using `Message.getIndex()`
+
+But we cannot obtain the message id by offset. Then we need to add a new API to get the message id by offset.
+
+# Motivation
+
Review Comment:
This is mentioned in Background knowledge. I have highlighted this sentence in bold. You can see whether it meets expectations or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org