[jira] [Updated] (SAMZA-882) Detect partition count changes in input streams

["Navina Ramesh (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Navina Ramesh updated SAMZA-882:
--------------------------------
    Attachment: SAMZA-882-0.patch

Attaching the diff. RB can be found here - https://reviews.apache.org/r/44187/


> Detect partition count changes in input streams
> -----------------------------------------------
>
>                 Key: SAMZA-882
>                 URL: https://issues.apache.org/jira/browse/SAMZA-882
>             Project: Samza
>          Issue Type: Bug
>    Affects Versions: 0.10.0
>            Reporter: Navina Ramesh
>            Assignee: Navina Ramesh
>             Fix For: 0.10.1
>
>         Attachments: SAMZA-882-0.patch
>
>
> This is a known issue where any change in the partition count in the upstream affects the Samza job and it needs to be restarted. In such scenarios, we experience data loss or incorrect processing because the application logic depends on the partitioning strategy. It is worsened by the fact that we don't even have a good mechanism to detect such a change. 
> As a first-step towards detection, I propose that we modify the stream metadata cache maintained in Samza such that when there a change in partition count, we increment a gauge metric. This way we can at least attach a hook to monitor when this happens and take necessary actions. 
> However, in the long-term, we need to come up with a better strategy for handling this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)