[jira] [Updated] (SAMZA-2724) [Elasticity] optimizations to improve throughput when elasticity is enabled by filtering out unwanted messages within SystemConsumers before RunLoop

["Lakshmi Manasa Gaduputi (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lakshmi Manasa Gaduputi updated SAMZA-2724:
-------------------------------------------
        Parent: SAMZA-2687
    Issue Type: Sub-task  (was: Improvement)

> [Elasticity] optimizations to improve throughput when elasticity is enabled by filtering out unwanted messages within SystemConsumers before RunLoop
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SAMZA-2724
>                 URL: https://issues.apache.org/jira/browse/SAMZA-2724
>             Project: Samza
>          Issue Type: Sub-task
>            Reporter: Lakshmi Manasa Gaduputi
>            Assignee: Lakshmi Manasa Gaduputi
>            Priority: Major
>
> When elasticity is enabled, the following optimizations can be done to improve throughput
>  
> in SystemConsumers, filter the messages which are not going to be processed by the RunLoop - aka of the messages fetched from consumer.poll(ssp) remove those messages which belong to the key buckets of the ssp not consumed by the container's job model. This will ensure RunLoop gets only those messages that it needs to process
>  
> Note that during prototyping this optimization, it was observed that this filtering causes a delay in the start of processing in all containers. this is due to all messages from an ssp being filtered out initially for ~7-10mins. This could be due to how the messages are fetched from the specific input topic. need a deeper investigation.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)