[jira] [Commented] (SAMZA-146) Throughput degrades unreasonably as the number of topic-partitions increases

["Jakob Homan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900544#comment-13900544 ] 

Jakob Homan commented on SAMZA-146:
-----------------------------------

In our production jobs that are consuming on the order of ~2500 topic partitions across four containers, we see performance drop significantly (by an order of magnitude).  This is particularly evident where most of the topics are relatively low volume compared to a few high volume topics.  Specifically,
{code:title=SystemConsumers.java|borderStyle=solid}
  def choose = {
    // ... contents elided
    refresh
    envelopeFromChooser
  }

  private def refresh {
    debug("Refreshing chooser with new messages.")

    // Poll every system for new messages.
    consumers.keys.foreach(poll(_))

    // Update the chooser.
    neededByChooser.foreach(systemStreamPartition =>
      // If we have messages for a stream that the chooser needs, then update.
      if (fetchMap(systemStreamPartition).intValue < maxMsgsPerStreamPartition) {
        chooser.update(unprocessedMessages(systemStreamPartition).dequeue)
        updateFetchMap(systemStreamPartition)
        neededByChooser -= systemStreamPartition
      })
  }
{code}
The call for each of the consumers to poll for new messages it expensive relative to the single call to choose (and subsequent delivery of a messages to process).  Instrumenting these calls shows that more than 99% of the calls to poll are returning back nothing.

> Throughput degrades unreasonably as the number of topic-partitions increases
> ----------------------------------------------------------------------------
>
>                 Key: SAMZA-146
>                 URL: https://issues.apache.org/jira/browse/SAMZA-146
>             Project: Samza
>          Issue Type: Bug
>          Components: container
>    Affects Versions: 0.7.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.7.0
>
>
> Currently we poll each stream partition in need of messages before each call to process.  For jobs with a large number of partitions, and particularly when those partitions are low volume, this dramatically impacts performance.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)