[jira] [Commented] (SAMZA-2531) Async Commit is Blocking for Window, Timer & End of Stream

["Daniel Chen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-2531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17221008#comment-17221008 ] 

Daniel Chen  commented on SAMZA-2531:
-------------------------------------

Thanks, will take a look

> Async Commit is Blocking for Window, Timer & End of Stream
> ----------------------------------------------------------
>
>                 Key: SAMZA-2531
>                 URL: https://issues.apache.org/jira/browse/SAMZA-2531
>             Project: Samza
>          Issue Type: Task
>            Reporter: Bharath Kumarasubramanian
>            Assignee: Daniel Chen 
>            Priority: Major
>
> We introduced non-blocking commit feature as part of  SAMZA-1140. Currently the commit is only non-blocking and allows process to be run concurrently.  
> As part of this task, we will need to evaluate what it takes to make it run concurrently along with window, timer & end of stream and if it is necessary in the first place. If not, rename the feature to explicitly callout process exclusive commit instead of async commit. 
> Some context:
>  * Transactional state currently doesn't work well w/ async commit turned on. i.e. Even though async commit, uses low watermark approach to find the offset to safely commit, it still doesn't guarantee the state snapshots happen during the commit correspond to the offset that is committed to checkpoint topic.
>  * Window, timer could potentially mutate the state in the same manner as process; So it might be okay to phase out (if we decide to support) the changes to introduce concurrency within window & timer first and follow it up with synchronizing the state w/ the offset that is committed. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)