[jira] [Updated] (KAFKA-9929) Support reverse iterator on WindowStore

["Matthias J. Sax (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-9929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias J. Sax updated KAFKA-9929:
-----------------------------------
    Labels: kip  (was: needs-kip)

> Support reverse iterator on WindowStore
> ---------------------------------------
>
>                 Key: KAFKA-9929
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9929
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Jorge Esteban Quilcate Otoya
>            Assignee: Jorge Esteban Quilcate Otoya
>            Priority: Major
>              Labels: kip
>             Fix For: 2.7.0
>
>
> Currently, WindowStore fetch operations return an iterator sorted from earliest to latest result:
> ```
> * For each key, the iterator guarantees ordering of windows, starting from the oldest/earliest
> * available window to the newest/latest window.
> ```
>  
> We have a use-case where traces are stored in a WindowStore<String, Trace> and use Kafka Streams to create a materialized view of traces. A query request comes with a time range (e.g. now-1h, now) and want to return the most recent results, i.e. fetch from this period of time, iterate and pattern match latest/most recent traces, and if enough results, then reply without moving further on the iterator.
> Same store is used to search for previous traces. In this case, it search a key for the last day, if found traces, we would also like to iterate from the most recent.
> RocksDb seems to support iterating backward and forward: [https://github.com/facebook/rocksdb/wiki/Iterator#iterating-upper-bound-and-lower-bound]
>  
> For reference: This in some way extracts some bits from this previous issue: https://issues.apache.org/jira/browse/KAFKA-4212:
>  
> > The {{RocksDBWindowsStore}}, a {{WindowsStore}}, can expire items via segment dropping, but it stores multiple items per key, based on their timestamp. But this store can be repurposed as a cache by fetching the items in reverse chronological order and returning the first item found.
>  
> Would like to know if there is any impediment on RocksDb or  WindowStore to support this.
> Adding an argument to reverse in current fetch methods would be great:
> ```
> WindowStore.fetch(from,to,Direction.BACKWARD|FORWARD)
> ```



--
This message was sent by Atlassian Jira
(v8.3.4#803005)