[jira] [Comment Edited] (KAFKA-4499) Add "getAllKeys" API for querying windowed KTable stores

["Richard Yu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16178445#comment-16178445 ] 

Richard Yu edited comment on KAFKA-4499 at 9/25/17 2:12 AM:
------------------------------------------------------------

So if I am understanding this correctly, the methods you have described above will be added to the {{WindowStore}} interface, and then be implemented in the {{ReadOnlyWindowStore}} class. 

Also, am I correct to assume that the changes in the underlying design which results from the addition of these methods will be restricted to the {{WindowStore}} interface and the {{ReadOnlyWindowStore}} implementations?


was (Author: yohan123):
So if I am understanding this correctly, the methods you have described above will be added to the {{WindowStore}} interface, and then be implemented in the {{ReadOnlyWindowStore}} class. 

Also, am I correct to assume that the changes in the underlying design which results from the addition of these methods will be restricted to the {{WindowStore}} interface and the {{ReadOnlyWindowStore}} class?

> Add "getAllKeys" API for querying windowed KTable stores
> --------------------------------------------------------
>
>                 Key: KAFKA-4499
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4499
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Matthias J. Sax
>              Labels: needs-kip
>
> Currently, both {{KTable}} and windowed-{{KTable}} stores can be queried via IQ feature. While {{ReadOnlyKeyValueStore}} (for {{KTable}} stores) provide method {{all()}} to scan the whole store (ie, returns an iterator over all stored key-value pairs), there is no similar API for {{ReadOnlyWindowStore}} (for windowed-{{KTable}} stores).
> This limits the usage of a windowed store, because the user needs to know what keys are stored in order the query it. It would be useful to provide possible APIs like this (only a rough sketch):
>  - {{keys()}} returns all keys available in the store (maybe together with available time ranges)
>  - {{all(long timeFrom, long timeTo)}} that returns all window for a specific time range
>  - {{allLatest()}} that returns the latest window for each key
> Because this feature would require to scan multiple segments (ie, RockDB instances) it would be quite inefficient with current store design. Thus, this feature also required to redesign the underlying window store itself.
> Because this is a major change, a KIP (https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals) is required. The KIP should cover the actual API design as well as the store refactoring.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)