Message ID for given event time

[Brian Candler <b....@pobox.com>]

When using the "reader" API, is there a way to get the message ID 
closest to a given event time?

That is, like "reset-cursor" on subscriptions, but for the reader API.  
At the moment I can only see how to start from the very beginning or the 
very end of a topic.

I couldn't find an existing FR on github, but I can open one if required.

Thanks,

Brian.

Re: Message ID for given event time

[Brian Candler <b....@pobox.com>]

On 02/11/2019 04:00, Sijie Guo wrote:
> Secondly, if you are looking for opening a Reader by publish time. I 
> don't think there is such a method in the "reader" API.
> Although it should be quite straightforward to add one in the 
> PulsarClient to open a reader by publish time. The implementation is 
> already in place.
> A reader is a consumer with non-durable subscription. The reader 
> implementation is based on the consumer implementation.
> Consumer already supports seek by publish time. We can just expose 
> that to Reader as well.
>
> Finally, contributions are very welcome!

My contribution is a github feature request: 
https://github.com/apache/pulsar/issues/5537

Java is not my thing, sorry :-)

Re: Message ID for given event time

[Sijie Guo <gu...@gmail.com>]

On Mon, Nov 4, 2019 at 11:05 PM Brian Candler <b....@pobox.com> wrote:

> On 02/11/2019 04:00, Sijie Guo wrote:
> > Currently there is no operations around event time.
> > The closest we have offered in Pulsar is seeking (or rest-cursor) by
> > publish time. Although the API is still a best-effort by doing binary
> > lookup.
>
> That reminds me of another question I meant to ask.  When a message is
> published, is the publish time assigned by the client or by the broker?


Currently it is assigned by the client.



>
> And does Pulsar do anything to ensure that publish time is monotonically
> increasing on a given topic?
>

Currently not. So I said it is a best-effort and doesn't offer any
guarantees.

We have talked about adding a broker-side timestamp and ensure it is
monotonically increasing. However it is not done yet.


>
> Since timestamps are at millisecond resolution, and you have two clients
> or two brokers with clocks that aren't properly synchronised, ISTM you
> could get values which aren't monotonic.
>
> Thanks,
>
> Brian.
>
>

Re: Message ID for given event time

[Brian Candler <b....@pobox.com>]

On 02/11/2019 04:00, Sijie Guo wrote:
> Currently there is no operations around event time.
> The closest we have offered in Pulsar is seeking (or rest-cursor) by 
> publish time. Although the API is still a best-effort by doing binary 
> lookup.

That reminds me of another question I meant to ask.  When a message is 
published, is the publish time assigned by the client or by the broker?  
And does Pulsar do anything to ensure that publish time is monotonically 
increasing on a given topic?

Since timestamps are at millisecond resolution, and you have two clients 
or two brokers with clocks that aren't properly synchronised, ISTM you 
could get values which aren't monotonic.

Thanks,

Brian.

Re: Message ID for given event time

[Sijie Guo <gu...@gmail.com>]

Brian,

First of all, I would like to clarify a bit about event time and publish
time. Currently there is no operations around event time.
The closest we have offered in Pulsar is seeking (or rest-cursor) by
publish time. Although the API is still a best-effort by doing binary
lookup.

Secondly, if you are looking for opening a Reader by publish time. I don't
think there is such a method in the "reader" API.
Although it should be quite straightforward to add one in the PulsarClient
to open a reader by publish time. The implementation is already in place.
A reader is a consumer with non-durable subscription. The reader
implementation is based on the consumer implementation.
Consumer already supports seek by publish time. We can just expose that to
Reader as well.

Finally, contributions are very welcome!

- Sijie

On Fri, Nov 1, 2019 at 8:12 PM Brian Candler <b....@pobox.com> wrote:

> When using the "reader" API, is there a way to get the message ID
> closest to a given event time?
>
> That is, like "reset-cursor" on subscriptions, but for the reader API.
> At the moment I can only see how to start from the very beginning or the
> very end of a topic.
>
> I couldn't find an existing FR on github, but I can open one if required.
>
> Thanks,
>
> Brian.
>
>