Multiple partitions per thread behavior

[Ian Friedman <ia...@flurry.com>]

Hey gang, quick consumer behavior question:

If I have a Consumer that owns several partitions but only creates a single stream for them, in what order will that stream get its messages? From my experiments it seems like the Consumer fetches some amount of messages for all N partitions it holds, and then processes all the messages for Stream 1, then Stream 2, etc. until it has run out of messages, then fetches more. Is that accurate?

--Ian

Re: Multiple partitions per thread behavior

[Guozhang Wang <wa...@gmail.com>]

Hi Ian,

That is correct. The consumer fetcher threads get data from multiple
partitions in batches, and put batches in to the stream one-by-one, hence
when stream iterates, it will likely give a sequence of messages from one
partition, then from another one, and so on.

Guozhang


On Thu, May 1, 2014 at 4:14 PM, Ian Friedman <ia...@flurry.com> wrote:

> Hey gang, quick consumer behavior question:
>
> If I have a Consumer that owns several partitions but only creates a
> single stream for them, in what order will that stream get its messages?
> From my experiments it seems like the Consumer fetches some amount of
> messages for all N partitions it holds, and then processes all the messages
> for Stream 1, then Stream 2, etc. until it has run out of messages, then
> fetches more. Is that accurate?
>
> --Ian




-- 
-- Guozhang