You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Shanthoosh Venkataraman (JIRA)" <ji...@apache.org> on 2019/01/25 23:26:00 UTC
[jira] [Updated] (SAMZA-2088) Update starting offset of a SSP in
OffsetManager to the startpoint.
[ https://issues.apache.org/jira/browse/SAMZA-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shanthoosh Venkataraman updated SAMZA-2088:
-------------------------------------------
Summary: Update starting offset of a SSP in OffsetManager to the startpoint. (was: Make TaskInstance work with Startpoints)
> Update starting offset of a SSP in OffsetManager to the startpoint.
> -------------------------------------------------------------------
>
> Key: SAMZA-2088
> URL: https://issues.apache.org/jira/browse/SAMZA-2088
> Project: Samza
> Issue Type: New Feature
> Reporter: Shanthoosh Venkataraman
> Assignee: Shanthoosh Venkataraman
> Priority: Major
>
> TaskInstance receives input messages from the AsyncRunLoop(i.e, event loop that coordinates the input message dispatch to the tasks within a SamzaContainer) and routes them to the user-defined task implementation.
> Starting offset for a input SystemStreamPartition is calculated in samza as one offset ahead of the previous checkpointed offset of the SystemStreamPartition.
> Currently, TaskInstance is instantiated with the starting offsets as part of AsycnRunLoop initialization. TaskInstance does not route the message of a SSP to the user-defined task implementation when the offset of received message is lower than the starting offset of SSP.
> SAMZA-1985 introduced the startpoint abstraction, where during AsyncRunLoop initialization, the idea is to move the input consumer to an offset defined by the startpoint.
> However, moving the consumer to an offset back alone does not suffice. For instance, if the check-pointed offset is 100 for a SSP and the startpoint intends to move the consumer back to offset 0. Even if we successfully moved the consumer instance to offset 0, the messages from offset 0 to 99 will not be routed by TaskInstance to the user defined task implementation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)