[jira] [Commented] (SAMZA-677) Support changelog for stores with TTL

["Naveen Somasundaram (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14542892#comment-14542892 ] 

Naveen Somasundaram commented on SAMZA-677:
-------------------------------------------

1. One possible way is set the TTL to same value as the Kafka's retention time. Kafka will wipe out data with a fixed time, within that time interval it's OK to restore your store. 
2. We store some meta information with each key-value (preferably in the key - but this will cause log compaction to fail), so we need to think more about this in detail, but essentially a meta information as a part of the message itself. And we do a restore, we skip the ones which have expired and write a expired_key -> null, which will trigger log compaction and wipe the data out eventually. 

> Support changelog for stores with TTL
> -------------------------------------
>
>                 Key: SAMZA-677
>                 URL: https://issues.apache.org/jira/browse/SAMZA-677
>             Project: Samza
>          Issue Type: Improvement
>    Affects Versions: 0.10.0
>            Reporter: Naveen Somasundaram
>
> We support TTL for RocksDB in Samza, but what we don't support is TTL in the changelog itself. It becomes a little tricky because we don't have a way to know from the underlying store that the record has expired. This will enable durable storage for TTL based stores. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)