You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Chris Riccomini (JIRA)" <ji...@apache.org> on 2013/08/20 03:38:51 UTC
[jira] [Commented] (SAMZA-23) TaskInstance commits for all
TaskCoordinator.commit calls
[ https://issues.apache.org/jira/browse/SAMZA-23?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744605#comment-13744605 ]
Chris Riccomini commented on SAMZA-23:
--------------------------------------
One thing to be careful of here is the case where a single StreamTask inside the SamzaContainer is about to block for a long time. For example, you might have a StreamTask for partition 7 which is about to re-calculate model weights, which might take 1-5 minutes. In such a case, it would actually be better for the StreamTask to trigger a commit for all partitions in the SamzaContainer, since it will be blocking processing for all StreamTasks in the SamzaContainer.
I argue that it's not worth worrying about this case, because the StreamTask is about to block the whole SamzaContainer (and all partitions it's processing), anyway. If some messages are stuck in a SystemProducer buffer while the blocking operation happens, it's effectively no different than if the messages hadn't yet been processed. Plus, future incoming messages will be blocked due to the blocking operation, anyway. This style of processing is inherently going to introduce lag. Let's not worry about it.
> TaskInstance commits for all TaskCoordinator.commit calls
> ---------------------------------------------------------
>
> Key: SAMZA-23
> URL: https://issues.apache.org/jira/browse/SAMZA-23
> Project: Samza
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
>
> If a StreamTask calls TaskCoordinator.commit, all TaskInstances will commit their SystemProducers, TaskStorageManager, and CheckpointManager. The problem with this is that if you have 400 partitions in a SamzaContainer (for example), and each calls TaskCoordinator.commit once per second, you actually get 400 commits per TaskInstance per second. This is incorrect behavior. We should make TaskInstance commit itself only when its task.commit.ms window has expired, or when TaskCoordinator.commit was called ONLY by its StreamTask partition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira