You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Bharath Kumarasubramanian (JIRA)" <ji...@apache.org> on 2019/07/17 21:15:00 UTC
[jira] [Commented] (SAMZA-2274) ensure logged and non logged store
to use different path
[ https://issues.apache.org/jira/browse/SAMZA-2274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16887428#comment-16887428 ]
Bharath Kumarasubramanian commented on SAMZA-2274:
--------------------------------------------------
To clarify, logged store by extension are durable and are backed by Kafka. As long as the changelog wasn't created by the application explicitly, it should be log compacted topic by default and there shouldn't be any data loss.
I think warning users about the basic directory being the same for both stores is useful and educates the users to make conscious choices on logged and non-logged directories. We can update the configuration table to reorder configurations and call out explicitly the importance of setting these configurations for stateful applications to bring awareness.
> ensure logged and non logged store to use different path
> --------------------------------------------------------
>
> Key: SAMZA-2274
> URL: https://issues.apache.org/jira/browse/SAMZA-2274
> Project: Samza
> Issue Type: Bug
> Reporter: Hai Lu
> Assignee: Bharath Kumarasubramanian
> Priority: Major
>
> currently logged and non logged store ("job.logged.store.base.dir" and "job.non-logged.store.base.dir") return the same default values if not set. This is problematic because we always clean up non-logged store path when starting up, which would end up eliminating the logged store content as well and cause major data/state loss. The proposed fix here is:
> 1) logged and non logged store should pick different default values
> 2) we explicitly do a validation that these two configs are set to different; throw exceptions otherwise
> 3) update our config documentation about these two configs
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)