You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by SenthilKumar K <se...@gmail.com> on 2017/05/27 14:55:14 UTC
Efficient way of Searching Messages By Timestamp - Kafka
Hello Kafka Developers , Users ,
We are exploring the SearchMessageByTimestamp feature in Kafka for our
use case .
Use Case : Kafka will be realtime message bus , users should be able to
pull Logs by specifying start_date and end_date or Pull me last five
minutes data etc ...
I did POC on SearchMessageByTimestamp , here is the code
https://gist.github.com/senthilec566/16e8e28b32834666fea132afc3a4e2f9 . And
i observed that Searching Messages is slow ..
Here is small test i did :
Query :Fetch Logs of Last *5 minutes*:
Result:
No of Records fetched : *30*
Fetch Time *6210* ms
Above test performed in a topic which has 4 partitions. In each partition
search & query processing happened .. in other words
consumer.offsetsForTimes()
consumer.assign(Arrays.asList(partition))
consumer.seek(this.partition, offsetTimestamp.offset())
consumer.poll(100)
are the API calls of each partition.. I realized that , this was the reason
for Kafka taking more time..
What is efficient way of implementing SerachMessageByTimeStamp ? Is Kafka
right candidate for our Use Case ?
Pls add your thoughts here ...
Cheers,
Senthil
Re: Efficient way of Searching Messages By Timestamp - Kafka
Posted by SenthilKumar K <se...@gmail.com>.
Hi Dev, It would be great if anybody share your experience on Search
Message by Timestamp ..
Cheer's,
Senthil
On May 28, 2017 2:08 AM, "SenthilKumar K" <se...@gmail.com> wrote:
> Hi Team , Any help here Pls ?
>
> Cheers,
> Senthil
>
> On Sat, May 27, 2017 at 8:25 PM, SenthilKumar K <se...@gmail.com>
> wrote:
>
>> Hello Kafka Developers , Users ,
>>
>> We are exploring the SearchMessageByTimestamp feature in Kafka for
>> our use case .
>>
>> Use Case : Kafka will be realtime message bus , users should be able
>> to pull Logs by specifying start_date and end_date or Pull me last five
>> minutes data etc ...
>>
>> I did POC on SearchMessageByTimestamp , here is the code
>> https://gist.github.com/senthilec566/16e8e28b32834666fea132afc3a4e2f9 .
>> And i observed that Searching Messages is slow ..
>>
>> Here is small test i did :
>> Query :Fetch Logs of Last *5 minutes*:
>> Result:
>> No of Records fetched : *30*
>> Fetch Time *6210* ms
>>
>> Above test performed in a topic which has 4 partitions. In each partition
>> search & query processing happened .. in other words
>> consumer.offsetsForTimes()
>> consumer.assign(Arrays.asList(partition))
>> consumer.seek(this.partition, offsetTimestamp.offset())
>> consumer.poll(100)
>>
>> are the API calls of each partition.. I realized that , this was the
>> reason for Kafka taking more time..
>>
>> What is efficient way of implementing SerachMessageByTimeStamp ? Is
>> Kafka right candidate for our Use Case ?
>>
>> Pls add your thoughts here ...
>>
>>
>> Cheers,
>> Senthil
>>
>
>
Re: Efficient way of Searching Messages By Timestamp - Kafka
Posted by SenthilKumar K <se...@gmail.com>.
Hi Team , Any help here Pls ?
Cheers,
Senthil
On Sat, May 27, 2017 at 8:25 PM, SenthilKumar K <se...@gmail.com>
wrote:
> Hello Kafka Developers , Users ,
>
> We are exploring the SearchMessageByTimestamp feature in Kafka for our
> use case .
>
> Use Case : Kafka will be realtime message bus , users should be able
> to pull Logs by specifying start_date and end_date or Pull me last five
> minutes data etc ...
>
> I did POC on SearchMessageByTimestamp , here is the code
> https://gist.github.com/senthilec566/16e8e28b32834666fea132afc3a4e2f9 .
> And i observed that Searching Messages is slow ..
>
> Here is small test i did :
> Query :Fetch Logs of Last *5 minutes*:
> Result:
> No of Records fetched : *30*
> Fetch Time *6210* ms
>
> Above test performed in a topic which has 4 partitions. In each partition
> search & query processing happened .. in other words
> consumer.offsetsForTimes()
> consumer.assign(Arrays.asList(partition))
> consumer.seek(this.partition, offsetTimestamp.offset())
> consumer.poll(100)
>
> are the API calls of each partition.. I realized that , this was the
> reason for Kafka taking more time..
>
> What is efficient way of implementing SerachMessageByTimeStamp ? Is Kafka
> right candidate for our Use Case ?
>
> Pls add your thoughts here ...
>
>
> Cheers,
> Senthil
>
Re: Efficient way of Searching Messages By Timestamp - Kafka
Posted by SenthilKumar K <se...@gmail.com>.
Hi Team , Any help here Pls ?
Cheers,
Senthil
On Sat, May 27, 2017 at 8:25 PM, SenthilKumar K <se...@gmail.com>
wrote:
> Hello Kafka Developers , Users ,
>
> We are exploring the SearchMessageByTimestamp feature in Kafka for our
> use case .
>
> Use Case : Kafka will be realtime message bus , users should be able
> to pull Logs by specifying start_date and end_date or Pull me last five
> minutes data etc ...
>
> I did POC on SearchMessageByTimestamp , here is the code
> https://gist.github.com/senthilec566/16e8e28b32834666fea132afc3a4e2f9 .
> And i observed that Searching Messages is slow ..
>
> Here is small test i did :
> Query :Fetch Logs of Last *5 minutes*:
> Result:
> No of Records fetched : *30*
> Fetch Time *6210* ms
>
> Above test performed in a topic which has 4 partitions. In each partition
> search & query processing happened .. in other words
> consumer.offsetsForTimes()
> consumer.assign(Arrays.asList(partition))
> consumer.seek(this.partition, offsetTimestamp.offset())
> consumer.poll(100)
>
> are the API calls of each partition.. I realized that , this was the
> reason for Kafka taking more time..
>
> What is efficient way of implementing SerachMessageByTimeStamp ? Is Kafka
> right candidate for our Use Case ?
>
> Pls add your thoughts here ...
>
>
> Cheers,
> Senthil
>