[jira] [Created] (SAMZA-2393) Side input streams should respect offset reset and broadcast properties

["Bharath Kumarasubramanian (Jira)" <ji...@apache.org>]

Bharath Kumarasubramanian created SAMZA-2393:
------------------------------------------------

             Summary: Side input streams should respect offset reset and broadcast properties
                 Key: SAMZA-2393
                 URL: https://issues.apache.org/jira/browse/SAMZA-2393
             Project: Samza
          Issue Type: Improvement
            Reporter: Bharath Kumarasubramanian


Side inputs are treated like every other stream (mostly) in Samza today. However, some of the properties of the stream like offset.reset policies, broadcast etc don't apply to side inputs. As part of this ticket, we want to explore the option of supporting these for side input streams too.

The current limitations are we don't have checkpointing for side inputs and use the local checkpoint on the disk to register the starting offsets. We need to tie down the offset manager into side inputs consumers which is capable of handling offset reset policies. 

Also, side input consumer updates the chooser every time when it chooses the message before waiting for the messages to be processed by the side input processor. This limits us to use priorities in side inputs and also implement broadcast correctly. We would need to do something similar to the RunLoop to support the above. 

As part of this ticket, we need to explore if we can use a new run loop instance for side inputs or extract some of the runloop logic to make it applicable for side inputs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)