You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/09/29 04:00:45 UTC

[GitHub] [pulsar] Jason918 opened a new issue #12234: [PIP] Add seek by index feature for consumer

Jason918 opened a new issue #12234:
URL: https://github.com/apache/pulsar/issues/12234

## Motivation

Currently we can reset the read position of a cursor by message id or timestamp. Since we formerly introduced index in broker metadata since 2.8.0, reset cursor by index is very helpful in other protocol handler (KoP or RoP).

## Goal

Add seek by index feature for consumer.

## API Changes

Add following interface in `org.apache.pulsar.client.api.Consumer`
- void seekByIndex(long index)
- CompletableFuture<Void> seekByIndexAsync(long index);

Here is the docs for this new interface.
Reset the subscription associated with this consumer to a specific message index.
For example, giving the index of message in this topic is in range [A, B), where A <= B;
- if seekByIndex(X), where X in range [A,B), message with index a is the next message consumer will receive.
- if seekByIndex(X), where X < A, it's the same as Consumer#seek(MessageId.earliest). Reset the subscription on the earliest message available in the topic.
- if seekByIndex(X), where X >= B, it's the same as Consumer#seek(MessageId.latest). Reset the subscription on the latest message in the topic.

Note: "org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor" must be added to "brokerEntryMetadataInterceptors" in broker configuration to enable index meta in broker.

## Implementation

For client-broker communication, we can reuse the old CommandSeek, which is used for `seek(messageId)` and `seek(timestamp)`. Add an optional field "index" for this interface.

In broker, the implementation for seek by index is basically the same as seek by timestamp. The key is to find the messageId with the giving index. A new class PersistentMessageFinderByIndex is introduced to do this. The difference between PersistentMessageFinderByIndex and origin PersistentMessageFinder is that, PersistentMessageFinderByIndex use BrokerEntryMetadata#index of the searching message instead of BrokerEntryMetadata#brokerTimestamp. Once we find the messageId, we can reset the subscription with `PersistentSubscription#resetCursor`.

## Reject Alternatives

No alternatives yet.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Jason918 closed issue #12234: [PIP 101] Add seek by index feature for consumer

Posted by GitBox <gi...@apache.org>.

Jason918 closed issue #12234:
URL: https://github.com/apache/pulsar/issues/12234


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] BewareMyPower commented on issue #12234: [PIP] Add seek by index feature for consumer

Posted by GitBox <gi...@apache.org>.

BewareMyPower commented on issue #12234:
URL: https://github.com/apache/pulsar/issues/12234#issuecomment-929917416

Let's move the discussion here. @MarvinCai

> But not all users will use Index or even BrokerEntryMetadata, while all messages have Timestamp and MessageID, that's a bit different in my opinion. Adding a method that rely on a property might not exist for all users is kind of weird.

I think it's a document issue. If we've documented well for the `seekByIndex` method, there will be nothing confused for users. Just like we've exposed BrokerEntryMetadata in #11553, the added methods of `Message` also requires broker to enable BrokerEntryMetadata.

https://github.com/apache/pulsar/blob/879e93d07c36567f6ba7792cc9274142ff4c0d74/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Message.java#L274-L282

(Though I think this doc is not well. We should make it clear that which configs should be configured in broker.)

Similarly, The current seek operation by message id also has some restrictions.

https://github.com/apache/pulsar/blob/879e93d07c36567f6ba7792cc9274142ff4c0d74/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Consumer.java#L586-L587

In conclusion, it provides a solution for users that are eager for message count based seek semantics. At the same time, it doesn't affect existing APIs.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] eolivelli commented on issue #12234: [PIP 101] Add seek by index feature for consumer

Posted by GitBox <gi...@apache.org>.

eolivelli commented on issue #12234:
URL: https://github.com/apache/pulsar/issues/12234#issuecomment-932004008


   I have assigned PIP101 and added the link in the Wiki page
   https://github.com/apache/pulsar/wiki
   
   @merlimat @michaeljmarshall 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #12234: [PIP 101] Add seek by index feature for consumer

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on issue #12234:
URL: https://github.com/apache/pulsar/issues/12234#issuecomment-1053791774


   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org