Partitions per topic per broker?

[Ian Friedman <ia...@flurry.com>]

Hi guys, apologies in advance for the newb question: 

I am running a 3 broker setup, and I have a topic configured with 100 partitions in the broker config. But I've noticed that what seems to happen is that each broker gets 100 partitions and it looks kind of like this in the consumer logs: 1-0, 1-1, 1-2, ... 1-99, 2-0, 2-1... etc. So the topic actually has 300 partitions, 100 per broker? Why? How does this work with the random partitioning scheme? I don't really understand the design behind this and my reading of the documentation hasn't seemed to turn up anything (though it's entirely possible I missed it). 

Thanks in advance! 

-- 
Ian Friedman

Re: Partitions per topic per broker?

[Philip O'Toole <ph...@loggly.com>]

You set the partition-count to 100 per broker. 3 brokers. 300 partitions total.

Philip

On Thu, Jul 25, 2013 at 11:29 AM, Ian Friedman <ia...@flurry.com> wrote:
> Hi guys, apologies in advance for the newb question:
>
> I am running a 3 broker setup, and I have a topic configured with 100 partitions in the broker config. But I've noticed that what seems to happen is that each broker gets 100 partitions and it looks kind of like this in the consumer logs: 1-0, 1-1, 1-2, ... 1-99, 2-0, 2-1... etc. So the topic actually has 300 partitions, 100 per broker? Why? How does this work with the random partitioning scheme? I don't really understand the design behind this and my reading of the documentation hasn't seemed to turn up anything (though it's entirely possible I missed it).
>
> Thanks in advance!
>
> --
> Ian Friedman
>

Re: Partitions per topic per broker?

[Ian Friedman <ia...@flurry.com>]

Thanks for the clarification guys. 

-- 
Ian Friedman


On Friday, July 26, 2013 at 1:16 AM, Jun Rao wrote:

> I assume this is 0.7. In 0.7, partitions are per broker. So, if you
> configure 100 partitions, each broker will have 100 partitions. In 0.8,
> partitions is at the cluster level and won't change when new brokers are
> added.
> 
> Thanks,
> 
> Jun
> 
> 
> On Thu, Jul 25, 2013 at 11:29 AM, Ian Friedman <ian@flurry.com (mailto:ian@flurry.com)> wrote:
> 
> > Hi guys, apologies in advance for the newb question:
> > 
> > I am running a 3 broker setup, and I have a topic configured with 100
> > partitions in the broker config. But I've noticed that what seems to happen
> > is that each broker gets 100 partitions and it looks kind of like this in
> > the consumer logs: 1-0, 1-1, 1-2, ... 1-99, 2-0, 2-1... etc. So the topic
> > actually has 300 partitions, 100 per broker? Why? How does this work with
> > the random partitioning scheme? I don't really understand the design behind
> > this and my reading of the documentation hasn't seemed to turn up anything
> > (though it's entirely possible I missed it).
> > 
> > Thanks in advance!
> > 
> > --
> > Ian Friedman
> > 
> 
> 
> 

Re: Partitions per topic per broker?

[Jun Rao <ju...@gmail.com>]

I assume this is 0.7. In 0.7, partitions are per broker. So, if you
configure 100 partitions, each broker will have 100 partitions. In 0.8,
partitions is at the cluster level and won't change when new brokers are
added.

Thanks,

Jun


On Thu, Jul 25, 2013 at 11:29 AM, Ian Friedman <ia...@flurry.com> wrote:

> Hi guys, apologies in advance for the newb question:
>
> I am running a 3 broker setup, and I have a topic configured with 100
> partitions in the broker config. But I've noticed that what seems to happen
> is that each broker gets 100 partitions and it looks kind of like this in
> the consumer logs: 1-0, 1-1, 1-2, ... 1-99, 2-0, 2-1... etc. So the topic
> actually has 300 partitions, 100 per broker? Why? How does this work with
> the random partitioning scheme? I don't really understand the design behind
> this and my reading of the documentation hasn't seemed to turn up anything
> (though it's entirely possible I missed it).
>
> Thanks in advance!
>
> --
> Ian Friedman
>
>