Best practices for changing partition numbers

[David Ross <dy...@klout.com>]

Hello,

We have found that, for our application, having a number of total
partitions as a multiple of the number of consumer hosts is beneficial.
Because of this, whenever we add or remove consumer hosts, we have to
change the number of partitions in the server config.

What are best practices for changing the number of partitions? It seems
like adding partitions is fine but removing partitions would result in data
loss - am I right? Is that avoidable? Is it preferable to bring in new
servers with new partitions? Anything else I should keep in mind on this
issue?


Thanks!

David

Re: Best practices for changing partition numbers

[Jun Rao <ju...@gmail.com>]

Reducing # partitions is going to be tricky. The data for those dropped
partitions will just be lost.

Thanks,

Jun

On Tue, Jan 8, 2013 at 1:24 PM, David Ross <dy...@klout.com> wrote:

> Yeah that makes sense, but what if we do need to change the number of
> partitions? What if we need to reduce it?
>
> On Tue, Jan 8, 2013 at 12:42 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > If you don't have a lot of topics, one thing you can do is to
> > over-partition a topic.
> >
> > Also, in 0.7, # of partitions grows with brokers. This is going to change
> > in 0.8, in which # of partitions is specified at topic creation time and
> > won't change as brokers change. One needs to use an admin DDL to change #
> > of partitions.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 7, 2013 at 10:23 PM, David Ross <dy...@klout.com> wrote:
> >
> > > Hello,
> > >
> > > We have found that, for our application, having a number of total
> > > partitions as a multiple of the number of consumer hosts is beneficial.
> > > Because of this, whenever we add or remove consumer hosts, we have to
> > > change the number of partitions in the server config.
> > >
> > > What are best practices for changing the number of partitions? It seems
> > > like adding partitions is fine but removing partitions would result in
> > data
> > > loss - am I right? Is that avoidable? Is it preferable to bring in new
> > > servers with new partitions? Anything else I should keep in mind on
> this
> > > issue?
> > >
> > >
> > > Thanks!
> > >
> > > David
> > >
> >
>

Re: Best practices for changing partition numbers

[David Ross <dy...@klout.com>]

I know how to change. I'm wondering what the ramifications are (losing
data, etc.).

On Tue, Jan 8, 2013 at 1:41 PM, Corbin Hoenes <co...@tynt.com> wrote:

> David
>
> Sounds like there is an admin ddl you have to run on your existing topic
> to change the # after creation.
>
> Sent from my iPhone
>
> On Jan 8, 2013, at 2:24 PM, David Ross <dy...@klout.com> wrote:
>
> > Yeah that makes sense, but what if we do need to change the number of
> > partitions? What if we need to reduce it?
> >
> > On Tue, Jan 8, 2013 at 12:42 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> >> If you don't have a lot of topics, one thing you can do is to
> >> over-partition a topic.
> >>
> >> Also, in 0.7, # of partitions grows with brokers. This is going to
> change
> >> in 0.8, in which # of partitions is specified at topic creation time and
> >> won't change as brokers change. One needs to use an admin DDL to change
> #
> >> of partitions.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Mon, Jan 7, 2013 at 10:23 PM, David Ross <dy...@klout.com> wrote:
> >>
> >>> Hello,
> >>>
> >>> We have found that, for our application, having a number of total
> >>> partitions as a multiple of the number of consumer hosts is beneficial.
> >>> Because of this, whenever we add or remove consumer hosts, we have to
> >>> change the number of partitions in the server config.
> >>>
> >>> What are best practices for changing the number of partitions? It seems
> >>> like adding partitions is fine but removing partitions would result in
> >> data
> >>> loss - am I right? Is that avoidable? Is it preferable to bring in new
> >>> servers with new partitions? Anything else I should keep in mind on
> this
> >>> issue?
> >>>
> >>>
> >>> Thanks!
> >>>
> >>> David
> >>
>

Re: Best practices for changing partition numbers

[Corbin Hoenes <co...@tynt.com>]

David

Sounds like there is an admin ddl you have to run on your existing topic to change the # after creation.

Sent from my iPhone

On Jan 8, 2013, at 2:24 PM, David Ross <dy...@klout.com> wrote:

> Yeah that makes sense, but what if we do need to change the number of
> partitions? What if we need to reduce it?
> 
> On Tue, Jan 8, 2013 at 12:42 PM, Jun Rao <ju...@gmail.com> wrote:
> 
>> If you don't have a lot of topics, one thing you can do is to
>> over-partition a topic.
>> 
>> Also, in 0.7, # of partitions grows with brokers. This is going to change
>> in 0.8, in which # of partitions is specified at topic creation time and
>> won't change as brokers change. One needs to use an admin DDL to change #
>> of partitions.
>> 
>> Thanks,
>> 
>> Jun
>> 
>> On Mon, Jan 7, 2013 at 10:23 PM, David Ross <dy...@klout.com> wrote:
>> 
>>> Hello,
>>> 
>>> We have found that, for our application, having a number of total
>>> partitions as a multiple of the number of consumer hosts is beneficial.
>>> Because of this, whenever we add or remove consumer hosts, we have to
>>> change the number of partitions in the server config.
>>> 
>>> What are best practices for changing the number of partitions? It seems
>>> like adding partitions is fine but removing partitions would result in
>> data
>>> loss - am I right? Is that avoidable? Is it preferable to bring in new
>>> servers with new partitions? Anything else I should keep in mind on this
>>> issue?
>>> 
>>> 
>>> Thanks!
>>> 
>>> David
>> 

Re: Best practices for changing partition numbers

[David Ross <dy...@klout.com>]

Yeah that makes sense, but what if we do need to change the number of
partitions? What if we need to reduce it?

On Tue, Jan 8, 2013 at 12:42 PM, Jun Rao <ju...@gmail.com> wrote:

> If you don't have a lot of topics, one thing you can do is to
> over-partition a topic.
>
> Also, in 0.7, # of partitions grows with brokers. This is going to change
> in 0.8, in which # of partitions is specified at topic creation time and
> won't change as brokers change. One needs to use an admin DDL to change #
> of partitions.
>
> Thanks,
>
> Jun
>
> On Mon, Jan 7, 2013 at 10:23 PM, David Ross <dy...@klout.com> wrote:
>
> > Hello,
> >
> > We have found that, for our application, having a number of total
> > partitions as a multiple of the number of consumer hosts is beneficial.
> > Because of this, whenever we add or remove consumer hosts, we have to
> > change the number of partitions in the server config.
> >
> > What are best practices for changing the number of partitions? It seems
> > like adding partitions is fine but removing partitions would result in
> data
> > loss - am I right? Is that avoidable? Is it preferable to bring in new
> > servers with new partitions? Anything else I should keep in mind on this
> > issue?
> >
> >
> > Thanks!
> >
> > David
> >
>

Re: Best practices for changing partition numbers

[Jun Rao <ju...@gmail.com>]

If you don't have a lot of topics, one thing you can do is to
over-partition a topic.

Also, in 0.7, # of partitions grows with brokers. This is going to change
in 0.8, in which # of partitions is specified at topic creation time and
won't change as brokers change. One needs to use an admin DDL to change #
of partitions.

Thanks,

Jun

On Mon, Jan 7, 2013 at 10:23 PM, David Ross <dy...@klout.com> wrote:

> Hello,
>
> We have found that, for our application, having a number of total
> partitions as a multiple of the number of consumer hosts is beneficial.
> Because of this, whenever we add or remove consumer hosts, we have to
> change the number of partitions in the server config.
>
> What are best practices for changing the number of partitions? It seems
> like adding partitions is fine but removing partitions would result in data
> loss - am I right? Is that avoidable? Is it preferable to bring in new
> servers with new partitions? Anything else I should keep in mind on this
> issue?
>
>
> Thanks!
>
> David
>