[jira] [Updated] (SAMZA-146) Throughput degrades unreasonably as the number of topic-partitions increases

["Jakob Homan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated SAMZA-146:
------------------------------

    Attachment: SAMZA-146.patch

Patch that adds a (capped) doubling back off to the calls to refresh, based on the result of the previous call.  This allows us to avoid the cost of polling each needed system every time and still be responsive when new messages are found.

In production this increased the throughput of the topic heavy jobs 4-5x and dropped the no-results-returned-from-polling rate down to around 70%.

The maximum back off is sensitive to the number of topics the job is consuming and is therefore made proportional to that.  Jobs consuming from just a few topics have a small backoff.  Testing against TestSamzaContainerTest.html shows improvement in jobs with large numbers of topic partitions and no impact for ones with small numbers.  This test doesn't simulate the distribution of message delivery rate described above and seen in production.  With that type of distribution, the improvement is much more dramatic, as described above.

> Throughput degrades unreasonably as the number of topic-partitions increases
> ----------------------------------------------------------------------------
>
>                 Key: SAMZA-146
>                 URL: https://issues.apache.org/jira/browse/SAMZA-146
>             Project: Samza
>          Issue Type: Bug
>          Components: container
>    Affects Versions: 0.7.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.7.0
>
>         Attachments: SAMZA-146.patch
>
>
> Currently we poll each stream partition in need of messages before each call to process.  For jobs with a large number of partitions, and particularly when those partitions are low volume, this dramatically impacts performance.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)