[jira] [Created] (KAFKA-8027) Gradual decline in performance of CachingWindowStore provider when number of keys grow

["Prashant (JIRA)" <ji...@apache.org>]

Prashant created KAFKA-8027:
-------------------------------

             Summary: Gradual decline in performance of CachingWindowStore provider when number of keys grow
                 Key: KAFKA-8027
                 URL: https://issues.apache.org/jira/browse/KAFKA-8027
             Project: Kafka
          Issue Type: Bug
          Components: streams
    Affects Versions: 2.1.0
            Reporter: Prashant


We observed this during a performance test of our stream application which tracks user's activity and provides REST interface to query the window state store.  We used default configuration of Materialized i.e. withCachingEnabled for storing user behaviour stats in a window state store (CompositeWindowStore with CachingWindowStore as underlyin which internally uses RocksDBStore for persistent).  

While querying window store with store.fetch(key, long, long), it internally tries to fetch the range from ThreadCache which uses a byte iterator to search for a key in cache and on a cache miss it goes to RocksDBStore for result. So, when number of keys in cache becomes large this ThreadCache search starts taking time (range Iterator on all keys) which impacts WindowStore query performance.

Workaround: If we disable cache with switch on Materialized instance i.e. withCachingDisabled, key search is delegated directly to RocksDBStore which is way faster and completed search in microseconds against millis in case of CachingWindowStore.  

Stats: With Unique users > 0.5M, random search for a key i.e. UserId:

 

withCachingEnabled :  40 < t < 80ms (upper bound increases as unique users grow) withCahingDisabled: t < 1ms (Almost constant time)      



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)