[jira] [Created] (SAMZA-2356) Do not trim changelog if time since last checkpoint is greater than min.compaction.lag

["Prateek Maheshwari (Jira)" <ji...@apache.org>]

Prateek Maheshwari created SAMZA-2356:
-----------------------------------------

             Summary: Do not trim changelog if time since last checkpoint is greater than min.compaction.lag
                 Key: SAMZA-2356
                 URL: https://issues.apache.org/jira/browse/SAMZA-2356
             Project: Samza
          Issue Type: Task
            Reporter: Prateek Maheshwari
            Assignee: Prateek Maheshwari


In the current implementation for transactional state, if the checkpointed changelog offset gets compacted, (e.g. if the processor was down for more than min compaction lag), and the checkpointed changelog offset is b/w the oldest and newest post compaction, we will end up deleting part of the current state (potentially everything after checkpointed offset). We should not do this.

Instead, we should write the checkpoint timestamp along with the the checkpointed changelog offset, and not do a trim during restore if time since last checkpoint is older than (or close to) min.compaction.lag. We should instead restore to latest and log a warning that transactional state guarantee doesn't hold (i.e., same as the current behavior). The fix for users would then be to increase the min.compaction.lag.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)