You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@samza.apache.org by "Lakshmi Manasa Gaduputi (Jira)" <ji...@apache.org> on 2021/09/15 23:09:00 UTC

[jira] [Created] (SAMZA-2687) Elasticity: scale up task count beyond the input partition count.

Lakshmi Manasa Gaduputi created SAMZA-2687:
----------------------------------------------

             Summary: Elasticity: scale up task count beyond the input partition count.
                 Key: SAMZA-2687
                 URL: https://issues.apache.org/jira/browse/SAMZA-2687
             Project: Samza
          Issue Type: New Feature
            Reporter: Lakshmi Manasa Gaduputi
            Assignee: Lakshmi Manasa Gaduputi


Problem: Throughput via parallelism is tied to the number of tasks which is equal to the partition count of input streams. If a job is facing lag and is already at the max container count = number of tasks = number of input partitions, then the only choice it is left with is to repartition the input. The need for this arises due to the process-time of the job’s logic which is not under Samza’s control.

Solution: Proposed approach is to allow consumption of a portion of the partition (SystemStreamPartition) by a virtual task. Virtual task is the same as a task except that it consumes sub-ssp. 

SEP to follow shortly

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)