[GitHub] kafka pull request #2159: KAFKA 4273 - Add TTL support for RocksDB

[dpoldrugo <gi...@git.apache.org>]

GitHub user dpoldrugo opened a pull request:

    https://github.com/apache/kafka/pull/2159

    KAFKA 4273 - Add TTL support for RocksDB

    Since Streams DSL doesn't support fine grained configurations of state stores (it usese only RocksDB) - I have added  new StreamsConfig called `rocksdb.ttl.sec` - which allows you to set TTL for all state stores used by the topology. To make short, if you set property to a value `>=1`, it will use TtlDB instead of RocksDB and this will lead to records getting expired after this defined period.
    
    This should help users to bound their disk usage and provide a configuration for use cases where your data has natural TTL/retention. For example, when you process data only for one hour, and after that you don't need the data in state stores anymore.
    
    I have added [test](https://github.com/apache/kafka/compare/trunk...dpoldrugo:KAFKA-4273-ttl-support?expand=1#diff-d908a80c770d196ac823752da3b3a864R117) to check if TtlDB is expiring record, but I can't make TtlDB expire record within a reasonable windows (1 minute). Do you have any suggestions how to force TtlDB to expire records more quickly?
    
    Since I'm using Kafka and Kafka Streams 0.10.1.0, I have also added this code to the [0.10.1](https://github.com/dpoldrugo/kafka/tree/0.10.1-KAFKA-4273-ttl-support) branch, and if the review goes well I hope it can be added to the 0.10.1.1 release.
    The patch is here: [KAFKA_4273_Add_TTL_support_for_RocksDB_v2.patch.txt](https://github.com/apache/kafka/files/607638/KAFKA_4273_Add_TTL_support_for_RocksDB_v2.patch.txt)
    
    **Suggestion for future work**
    Since this config/feature applies to all state stores, it would be nice to provide an API for users to configure TTL for every state store, for example during toplogy building with KStreamBuilder.
    
    Now: KStreamBuilder#table(String topic, final String storeName)
    Suggestion: KStreamBuilder#table(String topic, final String storeName, **_StoreOptions_ storeOptions**)
    Where **_StoreOptions_** would be something like this: `{ ttlSeconds: int }`
    
    More details: [KAFKA-4273](https://issues.apache.org/jira/browse/KAFKA-4273)
    
    @guozhangwang @dguy @mjsax @norwood @enothereska @ijuma    - could you check this out?

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dpoldrugo/kafka KAFKA-4273-ttl-support

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/2159.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2159
    
----
commit 5a3f1372daf2a0e939b246756c7e712e9ea21662
Author: dpoldrugo <da...@infobip.com>
Date:   2016-11-22T21:01:02Z

    KAFKA 4273 - Add TTL support for RocksDB

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---