[jira] [Updated] (SAMZA-2356) Do not trim changelog if time since last checkpoint is greater than min.compaction.lag

["Hai Lu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hai Lu updated SAMZA-2356:
--------------------------
    Fix Version/s: 1.3

> Do not trim changelog if time since last checkpoint is greater than min.compaction.lag
> --------------------------------------------------------------------------------------
>
>                 Key: SAMZA-2356
>                 URL: https://issues.apache.org/jira/browse/SAMZA-2356
>             Project: Samza
>          Issue Type: Task
>            Reporter: Prateek Maheshwari
>            Assignee: Prateek Maheshwari
>            Priority: Major
>             Fix For: 1.3
>
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> In the current implementation for transactional state, if the checkpointed changelog offset gets compacted, (e.g. if the processor was down for more than min compaction lag), and the checkpointed changelog offset is b/w the oldest and newest post compaction, we will end up deleting part of the current state (potentially everything after checkpointed offset). We should not do this.
> Instead, we should write the checkpoint timestamp along with the the checkpointed changelog offset, and not do a trim during restore if time since last checkpoint is older than (or close to) min.compaction.lag. We should instead restore to latest and log a warning that transactional state guarantee doesn't hold (i.e., same as the current behavior). The fix for users would then be to increase the min.compaction.lag.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)