Re: querying messages based on timestamps

[Zaiming Shi <zm...@gmail.com>]

Hi Jiangjie !
Does the word 'Currently' imply any plan in introducing timestamp in log
entries?

Regards
/Zaiming
On Jun 30, 2015 11:08 PM, "Jiangjie Qin" <jq...@linkedin.com.invalid> wrote:

> Currently Kafka only have a very coarse solution to find offset by time,
> which is based on the segment last modified time.
> This interface is only available in simple consumer. You may issue an
> OffsetRequest to specify a timestamp. The offset returned will be the
> first offset of segments whose last modification time is earlier than the
> timestamp you provided.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On 6/30/15, 1:16 PM, "Adam Dubiel" <du...@gmail.com> wrote:
>
> >We faced similar problem and ended up with implementing variant of golden
> >section search, that reads message using simple consumer and checks the
> >timestamp (timestamps are appended by our producer though, they do not
> >come
> >from any Kafka metadata) till it finds message closest to given date.
> >
> >Adam
> >
> >2015-06-30 21:52 GMT+02:00 Shushant Arora <sh...@gmail.com>:
> >
> >> Is it possible using low level consumer to get kafka messages based on
> >> timestamp, say I want to get all messages from last 5 minutes. I don't
> >>know
> >> what were offsets of partitions 5 minutes back.
> >>
> >> In low level consumer : when I gave epoch for  whichTime , it failed.
> >>  requestInfo.put(topicAndPartition, new
> >> PartitionOffsetRequestInfo(whichTime,
> >> 1));
> >>
> >> Is only latest and earliest supported in timestamp,Is there any way to
> >> filter messages based on timestamp?
> >>
> >> Thanks
> >> Shushant
> >>
>
>

Re: querying messages based on timestamps

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

Yes, we have plan to add better support for this.

Thanks,

Jiangjie (Becket) Qin

On 6/30/15, 4:23 PM, "Zaiming Shi" <zm...@gmail.com> wrote:

>Hi Jiangjie !
>Does the word 'Currently' imply any plan in introducing timestamp in log
>entries?
>
>Regards
>/Zaiming
>On Jun 30, 2015 11:08 PM, "Jiangjie Qin" <jq...@linkedin.com.invalid>
>wrote:
>
>> Currently Kafka only have a very coarse solution to find offset by time,
>> which is based on the segment last modified time.
>> This interface is only available in simple consumer. You may issue an
>> OffsetRequest to specify a timestamp. The offset returned will be the
>> first offset of segments whose last modification time is earlier than
>>the
>> timestamp you provided.
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>> On 6/30/15, 1:16 PM, "Adam Dubiel" <du...@gmail.com> wrote:
>>
>> >We faced similar problem and ended up with implementing variant of
>>golden
>> >section search, that reads message using simple consumer and checks the
>> >timestamp (timestamps are appended by our producer though, they do not
>> >come
>> >from any Kafka metadata) till it finds message closest to given date.
>> >
>> >Adam
>> >
>> >2015-06-30 21:52 GMT+02:00 Shushant Arora <sh...@gmail.com>:
>> >
>> >> Is it possible using low level consumer to get kafka messages based
>>on
>> >> timestamp, say I want to get all messages from last 5 minutes. I
>>don't
>> >>know
>> >> what were offsets of partitions 5 minutes back.
>> >>
>> >> In low level consumer : when I gave epoch for  whichTime , it failed.
>> >>  requestInfo.put(topicAndPartition, new
>> >> PartitionOffsetRequestInfo(whichTime,
>> >> 1));
>> >>
>> >> Is only latest and earliest supported in timestamp,Is there any way
>>to
>> >> filter messages based on timestamp?
>> >>
>> >> Thanks
>> >> Shushant
>> >>
>>
>>

Re: querying messages based on timestamps

[Bhavesh Mistry <mi...@gmail.com>]

We had similar requirement to re-load the data based on timestamp (range
between 1PM to 2PM) etc.

We store the relationship between timestamp and largest offset number in
Time Series Database using jmxtrans (LogEndOffset JMX bean vs current
time.).  You can setup polling interval to be 60 minutes via jmxtrans and
store the LogEndOffset offset (for each partitions).

Use this info from TSDB to fetch range offset any time ( last five minute
etc) for each partition.

Here is JMS bean info:
Log.LogEndOffset
Log.LogStartOffset
Log.size


I hope this helps.

Thanks,

Bhavesh



On Tue, Jun 30, 2015 at 4:23 PM, Zaiming Shi <zm...@gmail.com> wrote:

> Hi Jiangjie !
> Does the word 'Currently' imply any plan in introducing timestamp in log
> entries?
>
> Regards
> /Zaiming
> On Jun 30, 2015 11:08 PM, "Jiangjie Qin" <jq...@linkedin.com.invalid>
> wrote:
>
> > Currently Kafka only have a very coarse solution to find offset by time,
> > which is based on the segment last modified time.
> > This interface is only available in simple consumer. You may issue an
> > OffsetRequest to specify a timestamp. The offset returned will be the
> > first offset of segments whose last modification time is earlier than the
> > timestamp you provided.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On 6/30/15, 1:16 PM, "Adam Dubiel" <du...@gmail.com> wrote:
> >
> > >We faced similar problem and ended up with implementing variant of
> golden
> > >section search, that reads message using simple consumer and checks the
> > >timestamp (timestamps are appended by our producer though, they do not
> > >come
> > >from any Kafka metadata) till it finds message closest to given date.
> > >
> > >Adam
> > >
> > >2015-06-30 21:52 GMT+02:00 Shushant Arora <sh...@gmail.com>:
> > >
> > >> Is it possible using low level consumer to get kafka messages based on
> > >> timestamp, say I want to get all messages from last 5 minutes. I don't
> > >>know
> > >> what were offsets of partitions 5 minutes back.
> > >>
> > >> In low level consumer : when I gave epoch for  whichTime , it failed.
> > >>  requestInfo.put(topicAndPartition, new
> > >> PartitionOffsetRequestInfo(whichTime,
> > >> 1));
> > >>
> > >> Is only latest and earliest supported in timestamp,Is there any way to
> > >> filter messages based on timestamp?
> > >>
> > >> Thanks
> > >> Shushant
> > >>
> >
> >
>