[jira] [Commented] (KAFKA-6711) Checkpoint of InMemoryStore along with GlobalKTable

["Guozhang Wang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414308#comment-16414308 ] 

Guozhang Wang commented on KAFKA-6711:
--------------------------------------

Hello [~cemo], it is by-design that in-memory state stores will not persist any data to the local storage engine upon closing, and hence when restarting it will lose all the data and hence will need to re-bootstrap from the topic again. For normal state stores we are having the check that {{if (store.persistent() && storeToChangelogTopic.containsKey(storeName)) // write checkpoint values}}, but I agree with you that in global store implementation, i.e. {{GlobalStateManagerImpl}} we do not have this check, which I think is a bug. I.e. we should do the similar check and hence do not write offsets in the checkpoint file. Would you like to fix this issue?

As for your own observed issue, we do have a JIRA open for adding the checkpoint feature for in-memory stores so that data will not be lost upon closing: https://issues.apache.org/jira/browse/KAFKA-3184.

> Checkpoint of InMemoryStore along with GlobalKTable 
> ----------------------------------------------------
>
>                 Key: KAFKA-6711
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6711
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 1.0.1
>            Reporter: Cemalettin Koç
>            Priority: Major
>
> We are using an InMemoryStore along with GlobalKTable and I noticed that after each restart I am losing all my data. When I debug it, `/tmp/kafka-streams/category-client-1/global/.checkpoint` file contains offset for my GlobalKTable topic. I had checked GlobalStateManagerImpl implementation and noticed that it is not guarded for cases similar to mine. I am fairly new to Kafka land and probably there might be another way to fix issue. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)