[jira] [Commented] (SAMZA-1083) Do not load task stores which are older than delete tombstones during container startup

["Shanthoosh Venkataraman (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15857461#comment-15857461 ] 

Shanthoosh Venkataraman commented on SAMZA-1083:
------------------------------------------------

[~jmakes] : 

I've attached the patch with this ticket. 

Here is the RB : https://reviews.apache.org/r/52476/ associated with this change. This will delete the task stores older than delete.retention.ms during the container startup.

If a pull request has to be raised for this in github, I could do that as well. 

Thanks.

> Do not load task stores which are older than delete tombstones during container startup
> ---------------------------------------------------------------------------------------
>
>                 Key: SAMZA-1083
>                 URL: https://issues.apache.org/jira/browse/SAMZA-1083
>             Project: Samza
>          Issue Type: New Feature
>            Reporter: Shanthoosh Venkataraman
>            Assignee: Shanthoosh Venkataraman
>         Attachments: rb52476.patch
>
>
> Every local task store is backed up by a kafka changelog topic. Due to log compaction, delete tombstones of the changelog topic have a ttl of delete.retention.ms. Replaying the events from the changelog that has missing delete tombstones, would result in creation of an inconsistent local store(due to the missing of some delete events). This patch deletes the local stores in which difference between current time and last modified time of the offset file is greater than delete.retention.ms during the container startup.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)