[jira] [Updated] (KAFKA-3941) Avoid applying eviction listener in InMemoryKeyValueLoggedStore

["Guozhang Wang (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Guozhang Wang updated KAFKA-3941:
---------------------------------
    Description: 
This is reported by [~norwood].

In {{InMemoryKeyValueLoggedStore}} we set the eviction listener while creating the store, which records the evicted records as "deleted" in the changelogger, which will then send a tombstone record to the corresponding changelog topic partition. However, when restoring the store, although we are using the inner store's putInternal call and hence by-pass the logging since it is not needed, this eviction listener will still call the outer store's deleted call and hence still sends the tombstone record, causing the restoration process to fail, as it is not expecting the changelog log-end-offset to increase (i.e. more messages are appended to it) while restoration is going on.

We should defer the listener initialization until the end of the {{init}} call after the restoration is completed, and also making sure the "register" call is made at the inner stores only.

  was:
This is reported by [~norwood].

In {{InMemoryKeyValueLoggedStore}} we set the eviction listener which records the evicted records as deletes in the changelog. However, when restoring the store this listener will then double-writes the delete record, causing the restoration process to fail.

We should defer the listener initialization until the end of the {{init}} call, instead of inside the {{supplier.get}}.


> Avoid applying eviction listener in InMemoryKeyValueLoggedStore
> ---------------------------------------------------------------
>
>                 Key: KAFKA-3941
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3941
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>            Reporter: Guozhang Wang
>            Assignee: Guozhang Wang
>
> This is reported by [~norwood].
> In {{InMemoryKeyValueLoggedStore}} we set the eviction listener while creating the store, which records the evicted records as "deleted" in the changelogger, which will then send a tombstone record to the corresponding changelog topic partition. However, when restoring the store, although we are using the inner store's putInternal call and hence by-pass the logging since it is not needed, this eviction listener will still call the outer store's deleted call and hence still sends the tombstone record, causing the restoration process to fail, as it is not expecting the changelog log-end-offset to increase (i.e. more messages are appended to it) while restoration is going on.
> We should defer the listener initialization until the end of the {{init}} call after the restoration is completed, and also making sure the "register" call is made at the inner stores only.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)