You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/03/15 22:26:41 UTC
[jira] [Commented] (SAMZA-1146) TaskCallbackManager commit fix.
[ https://issues.apache.org/jira/browse/SAMZA-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15927096#comment-15927096 ]
ASF GitHub Bot commented on SAMZA-1146:
---------------------------------------
GitHub user shanthoosh opened a pull request:
https://github.com/apache/samza/pull/87
SAMZA-1146: TaskCallbackManager commit fix.
Each task callback in samza belongs to different SystemStreamPartition. When multiple callbacks in contagious order are available for commit, callback with highest sequence number is chosen for commit. This will prevent checkpointing of completed callbacks that has commit request and doesn't have highest sequence number. Upon task restart this will lead to duplicate reprocessing of already processed messages (since completed callbacks for some SystemStreamPartition's aren't committed earlier).
This PR fixes it and commits all completed callbacks that has commit request defined. Added a test to verify the behavior.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shanthoosh/samza Fixing_CallBackManager_Commit
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/87.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #87
----
commit 7afbbcef414bb2d3e64117a8e1880072e0a99471
Author: Shanthoosh Venkataraman <sv...@linkedin.com>
Date: 2017-03-09T19:54:20Z
SAMZA-1146 : TaskCallbackManager commit fix.
Changing TaskCallbackManager.update to return all completed callbacks that has
commit request defined for commit(not just the callback with highest sequence number).
----
> TaskCallbackManager commit fix.
> -------------------------------
>
> Key: SAMZA-1146
> URL: https://issues.apache.org/jira/browse/SAMZA-1146
> Project: Samza
> Issue Type: Bug
> Reporter: Shanthoosh Venkataraman
> Assignee: Shanthoosh Venkataraman
>
> Each task callback in samza belongs to different SystemStreamPartition. When multiple callbacks in contagious order are available for commit, callback with highest sequence number is chosen for commit. This will prevent checkpointing of completed callbacks that has commit request and doesn't have highest sequence number. Upon task restart this will lead to duplicate reprocessing of already processed messages (since completed callbacks for some SystemStreamPartition's aren't committed earlier).
> Expected behavior is to commit all completed callbacks that has commit request associated with it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)