You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by David Birdsong <da...@gmail.com> on 2014/05/21 10:23:44 UTC
reduce replication factor
Is there a way to reduce the replication count? I'm trying to spread
existing partitions across more brokers, but it's hard to decomm a broker.
Reducing repl count would suffice for now.
Any tips?
I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
Re: reduce replication factor
Posted by David Birdsong <da...@gmail.com>.
On Thu, May 22, 2014 at 7:29 AM, Jun Rao <ju...@gmail.com> wrote:
> That's exactly what partition reassignment does. It may take a bit of time
> for the reassignment to complete.
>
> For example, if you initiate the process to change the replica assignment
> from {1, 2, 3} to {4,5,6}, the values in the assigned replica (AR) and
> leader/isr path in ZK may go through the following transition.
> AR leader/isr
> 1. {1,2,3} 1/{1,2,3}
> 2. {1,2,3,4,5,6} 1/{1,2,3}
> 3. {1,2,3,4,5,6} 1/{1,2,3,4,5,6}
> 4. {1,2,3,4,5,6} 4/{1,2,3,4,5,6}
> 5. {1,2,3,4,5,6} 4/{4,5,6}
> 6. {4,5,6} 4/{4,5,6}
>
> Going from step 2 to step 3 can take some time since we have to wait for
> the new replicas to fully catch up. The reassignment process completes when
> you are in step 6. The reassignment tool has a mode that you can use to
> check the status.
>
> Thanks,
>
Great! This is progress.
So I think some of mucking around I was doing caused the broker that I
wanted to drop the partitions to lose it's state.
Re-running execute w/ a json file that listed my target replicas (already
synced by now) caused that broker to drop the partitions and leave the
replica set finally.
> Jun
>
>
>
> On Wed, May 21, 2014 at 10:09 PM, David Birdsong
> <da...@gmail.com>wrote:
>
> > On Wed, May 21, 2014 at 9:06 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Expansion can be done by following
> > > http://kafka.apache.org/documentation.html#basic_ops_cluster_expansion
> > >
> > > If you just want to free up a server, you can stop the broker there and
> > > start a broker using the same broker id on a new server. Data should be
> > > automatically replicated to the new server.
> > >
> > >
> > The thing is, I don't want to fully free up the server, I just want it to
> > stop hosting all the topic,partition sets it originally did.
> >
> > Here's the flow:
> >
> > broker_1 hosts topic,{1,2,3,4}
> > over time load for the partitions overwhelms broker_1
> >
> > spin up broker_2
> > migrate topic,{3,4} to broker_2 successfully
> >
> > broker_1 is only useful if partitions topic,{3,4} are dropped by broker_1
> >
> > how does one get broker_1 to disassociate, drop, forget, evict...(not
> sure
> > the verb) topic,{3,4} and let broker_2 own it.
> >
> > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, May 21, 2014 at 3:23 PM, David Birdsong <
> > david.birdsong@gmail.com
> > > >wrote:
> > >
> > > > Any suggestions? I'm kind of in a bind in that I don't understand how
> > to
> > > > grow the cluster when more capacity is needed--which happens to be
> > right
> > > > now for me.
> > > >
> > > > The only thing I can think that might work is to create new brokers,
> > > > cherry-pick topic/partition pairs and move them, then turn off the
> old
> > > ones
> > > > and forever retire their IDs freeing up my old hardware to come back
> > > online
> > > > as a new kafka broker ID.
> > > >
> > > >
> > > > On Wed, May 21, 2014 at 9:16 AM, David Birdsong <
> > > david.birdsong@gmail.com
> > > > >wrote:
> > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, May 21, 2014 at 9:11 AM, David Birdsong <
> > > > david.birdsong@gmail.com>wrote:
> > > > >
> > > > >> Here's the reassignment json and current "describe" output:
> > > > >> https://gist.github.com/davidbirdsong/32cd0c4f49496a6a32e5
> > > > >>
> > > > >>
> > > > >> In my re-assignment json, I tried to re-assign to 2 when the repl
> is
> > > set
> > > > >> to 3. Once I noticed the the completely new node "133" had
> appeared
> > in
> > > > the
> > > > >> ISR, I tried stopping 224, wiping kafka completely and then
> brought
> > > 224
> > > > >> back up as 224 again. It promptly replicated the topic, but never
> > > joined
> > > > >> the ISR.
> > > > >>
> > > > >
> > > > > to be clear, i don't want it to join the ISR. i'm curious how to
> make
> > > 224
> > > > > forget about the partitions for that topic since i have other plans
> > for
> > > > it.
> > > > >
> > > > >
> > > > >>
> > > > >> How does one move a replica? This is exactly what I'm trying to
> do.
> > > > >>
> > > > >> My pattern is a common one. I started with a set of 3 kafka
> brokers.
> > > The
> > > > >> load and space is overwhelming them. I'm trying to add new brokers
> > and
> > > > >> spread the partitions to new nodes while removing some of the
> > > > partitions on
> > > > >> the old nodes so as to make room. It's the latter that I don't get
> > how
> > > > to
> > > > >> do.
> > > > >>
> > > > >> I've conflated two issues here mostly due to needing to get this
> > > cluster
> > > > >> stable again.
> > > > >> - reduce replication
> > > > >> - remove a partition from a broker, ie. remove the replica
> > > > >>
> > > > >> they're very distinct actions, but both would help me in the
> moment
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Wed, May 21, 2014 at 8:56 AM, Jun Rao <ju...@gmail.com>
> wrote:
> > > > >>
> > > > >>> During the re-assignment, did you move the replica off the old
> > > broker?
> > > > >>>
> > > > >>> Thanks,
> > > > >>>
> > > > >>> Jun
> > > > >>>
> > > > >>>
> > > > >>> On Wed, May 21, 2014 at 8:21 AM, David Birdsong <
> > > > >>> david.birdsong@gmail.com>wrote:
> > > > >>>
> > > > >>> > I did that and so now the topic has 4 replicas for a repl count
> > of
> > > 3,
> > > > >>> but
> > > > >>> > only the 'new' replicas exist in the ISR.
> > > > >>> >
> > > > >>> > The old broker that I want to clear disk space and generally
> free
> > > up
> > > > >>> > resources has fully synced a topic that I want to disassociate
> > from
> > > > it.
> > > > >>> >
> > > > >>> > Is there a way to do this?
> > > > >>> >
> > > > >>> >
> > > > >>> > On Wed, May 21, 2014 at 7:48 AM, Jun Rao <ju...@gmail.com>
> > wrote:
> > > > >>> >
> > > > >>> > > We don't have an exact tool for doing this. You may be able
> to
> > do
> > > > >>> that
> > > > >>> > > through
> > > > >>> > >
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby
> > > > >>> > > specifying fewer replicas.
> > > > >>> > >
> > > > >>> > > Thanks,
> > > > >>> > >
> > > > >>> > > Jun
> > > > >>> > >
> > > > >>> > >
> > > > >>> > > On Wed, May 21, 2014 at 1:23 AM, David Birdsong <
> > > > >>> > david.birdsong@gmail.com
> > > > >>> > > >wrote:
> > > > >>> > >
> > > > >>> > > > Is there a way to reduce the replication count? I'm trying
> to
> > > > >>> spread
> > > > >>> > > > existing partitions across more brokers, but it's hard to
> > > decomm
> > > > a
> > > > >>> > > broker.
> > > > >>> > > > Reducing repl count would suffice for now.
> > > > >>> > > >
> > > > >>> > > > Any tips?
> > > > >>> > > >
> > > > >>> > > > I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
> > > > >>> > > >
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: reduce replication factor
Posted by Jun Rao <ju...@gmail.com>.
That's exactly what partition reassignment does. It may take a bit of time
for the reassignment to complete.
For example, if you initiate the process to change the replica assignment
from {1, 2, 3} to {4,5,6}, the values in the assigned replica (AR) and
leader/isr path in ZK may go through the following transition.
AR leader/isr
1. {1,2,3} 1/{1,2,3}
2. {1,2,3,4,5,6} 1/{1,2,3}
3. {1,2,3,4,5,6} 1/{1,2,3,4,5,6}
4. {1,2,3,4,5,6} 4/{1,2,3,4,5,6}
5. {1,2,3,4,5,6} 4/{4,5,6}
6. {4,5,6} 4/{4,5,6}
Going from step 2 to step 3 can take some time since we have to wait for
the new replicas to fully catch up. The reassignment process completes when
you are in step 6. The reassignment tool has a mode that you can use to
check the status.
Thanks,
Jun
On Wed, May 21, 2014 at 10:09 PM, David Birdsong
<da...@gmail.com>wrote:
> On Wed, May 21, 2014 at 9:06 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Expansion can be done by following
> > http://kafka.apache.org/documentation.html#basic_ops_cluster_expansion
> >
> > If you just want to free up a server, you can stop the broker there and
> > start a broker using the same broker id on a new server. Data should be
> > automatically replicated to the new server.
> >
> >
> The thing is, I don't want to fully free up the server, I just want it to
> stop hosting all the topic,partition sets it originally did.
>
> Here's the flow:
>
> broker_1 hosts topic,{1,2,3,4}
> over time load for the partitions overwhelms broker_1
>
> spin up broker_2
> migrate topic,{3,4} to broker_2 successfully
>
> broker_1 is only useful if partitions topic,{3,4} are dropped by broker_1
>
> how does one get broker_1 to disassociate, drop, forget, evict...(not sure
> the verb) topic,{3,4} and let broker_2 own it.
>
> Thanks,
> >
> > Jun
> >
> >
> > On Wed, May 21, 2014 at 3:23 PM, David Birdsong <
> david.birdsong@gmail.com
> > >wrote:
> >
> > > Any suggestions? I'm kind of in a bind in that I don't understand how
> to
> > > grow the cluster when more capacity is needed--which happens to be
> right
> > > now for me.
> > >
> > > The only thing I can think that might work is to create new brokers,
> > > cherry-pick topic/partition pairs and move them, then turn off the old
> > ones
> > > and forever retire their IDs freeing up my old hardware to come back
> > online
> > > as a new kafka broker ID.
> > >
> > >
> > > On Wed, May 21, 2014 at 9:16 AM, David Birdsong <
> > david.birdsong@gmail.com
> > > >wrote:
> > >
> > > >
> > > >
> > > >
> > > > On Wed, May 21, 2014 at 9:11 AM, David Birdsong <
> > > david.birdsong@gmail.com>wrote:
> > > >
> > > >> Here's the reassignment json and current "describe" output:
> > > >> https://gist.github.com/davidbirdsong/32cd0c4f49496a6a32e5
> > > >>
> > > >>
> > > >> In my re-assignment json, I tried to re-assign to 2 when the repl is
> > set
> > > >> to 3. Once I noticed the the completely new node "133" had appeared
> in
> > > the
> > > >> ISR, I tried stopping 224, wiping kafka completely and then brought
> > 224
> > > >> back up as 224 again. It promptly replicated the topic, but never
> > joined
> > > >> the ISR.
> > > >>
> > > >
> > > > to be clear, i don't want it to join the ISR. i'm curious how to make
> > 224
> > > > forget about the partitions for that topic since i have other plans
> for
> > > it.
> > > >
> > > >
> > > >>
> > > >> How does one move a replica? This is exactly what I'm trying to do.
> > > >>
> > > >> My pattern is a common one. I started with a set of 3 kafka brokers.
> > The
> > > >> load and space is overwhelming them. I'm trying to add new brokers
> and
> > > >> spread the partitions to new nodes while removing some of the
> > > partitions on
> > > >> the old nodes so as to make room. It's the latter that I don't get
> how
> > > to
> > > >> do.
> > > >>
> > > >> I've conflated two issues here mostly due to needing to get this
> > cluster
> > > >> stable again.
> > > >> - reduce replication
> > > >> - remove a partition from a broker, ie. remove the replica
> > > >>
> > > >> they're very distinct actions, but both would help me in the moment
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Wed, May 21, 2014 at 8:56 AM, Jun Rao <ju...@gmail.com> wrote:
> > > >>
> > > >>> During the re-assignment, did you move the replica off the old
> > broker?
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Jun
> > > >>>
> > > >>>
> > > >>> On Wed, May 21, 2014 at 8:21 AM, David Birdsong <
> > > >>> david.birdsong@gmail.com>wrote:
> > > >>>
> > > >>> > I did that and so now the topic has 4 replicas for a repl count
> of
> > 3,
> > > >>> but
> > > >>> > only the 'new' replicas exist in the ISR.
> > > >>> >
> > > >>> > The old broker that I want to clear disk space and generally free
> > up
> > > >>> > resources has fully synced a topic that I want to disassociate
> from
> > > it.
> > > >>> >
> > > >>> > Is there a way to do this?
> > > >>> >
> > > >>> >
> > > >>> > On Wed, May 21, 2014 at 7:48 AM, Jun Rao <ju...@gmail.com>
> wrote:
> > > >>> >
> > > >>> > > We don't have an exact tool for doing this. You may be able to
> do
> > > >>> that
> > > >>> > > through
> > > >>> > >
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby
> > > >>> > > specifying fewer replicas.
> > > >>> > >
> > > >>> > > Thanks,
> > > >>> > >
> > > >>> > > Jun
> > > >>> > >
> > > >>> > >
> > > >>> > > On Wed, May 21, 2014 at 1:23 AM, David Birdsong <
> > > >>> > david.birdsong@gmail.com
> > > >>> > > >wrote:
> > > >>> > >
> > > >>> > > > Is there a way to reduce the replication count? I'm trying to
> > > >>> spread
> > > >>> > > > existing partitions across more brokers, but it's hard to
> > decomm
> > > a
> > > >>> > > broker.
> > > >>> > > > Reducing repl count would suffice for now.
> > > >>> > > >
> > > >>> > > > Any tips?
> > > >>> > > >
> > > >>> > > > I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
> > > >>> > > >
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>
Re: reduce replication factor
Posted by David Birdsong <da...@gmail.com>.
On Wed, May 21, 2014 at 9:06 PM, Jun Rao <ju...@gmail.com> wrote:
> Expansion can be done by following
> http://kafka.apache.org/documentation.html#basic_ops_cluster_expansion
>
> If you just want to free up a server, you can stop the broker there and
> start a broker using the same broker id on a new server. Data should be
> automatically replicated to the new server.
>
>
The thing is, I don't want to fully free up the server, I just want it to
stop hosting all the topic,partition sets it originally did.
Here's the flow:
broker_1 hosts topic,{1,2,3,4}
over time load for the partitions overwhelms broker_1
spin up broker_2
migrate topic,{3,4} to broker_2 successfully
broker_1 is only useful if partitions topic,{3,4} are dropped by broker_1
how does one get broker_1 to disassociate, drop, forget, evict...(not sure
the verb) topic,{3,4} and let broker_2 own it.
Thanks,
>
> Jun
>
>
> On Wed, May 21, 2014 at 3:23 PM, David Birdsong <david.birdsong@gmail.com
> >wrote:
>
> > Any suggestions? I'm kind of in a bind in that I don't understand how to
> > grow the cluster when more capacity is needed--which happens to be right
> > now for me.
> >
> > The only thing I can think that might work is to create new brokers,
> > cherry-pick topic/partition pairs and move them, then turn off the old
> ones
> > and forever retire their IDs freeing up my old hardware to come back
> online
> > as a new kafka broker ID.
> >
> >
> > On Wed, May 21, 2014 at 9:16 AM, David Birdsong <
> david.birdsong@gmail.com
> > >wrote:
> >
> > >
> > >
> > >
> > > On Wed, May 21, 2014 at 9:11 AM, David Birdsong <
> > david.birdsong@gmail.com>wrote:
> > >
> > >> Here's the reassignment json and current "describe" output:
> > >> https://gist.github.com/davidbirdsong/32cd0c4f49496a6a32e5
> > >>
> > >>
> > >> In my re-assignment json, I tried to re-assign to 2 when the repl is
> set
> > >> to 3. Once I noticed the the completely new node "133" had appeared in
> > the
> > >> ISR, I tried stopping 224, wiping kafka completely and then brought
> 224
> > >> back up as 224 again. It promptly replicated the topic, but never
> joined
> > >> the ISR.
> > >>
> > >
> > > to be clear, i don't want it to join the ISR. i'm curious how to make
> 224
> > > forget about the partitions for that topic since i have other plans for
> > it.
> > >
> > >
> > >>
> > >> How does one move a replica? This is exactly what I'm trying to do.
> > >>
> > >> My pattern is a common one. I started with a set of 3 kafka brokers.
> The
> > >> load and space is overwhelming them. I'm trying to add new brokers and
> > >> spread the partitions to new nodes while removing some of the
> > partitions on
> > >> the old nodes so as to make room. It's the latter that I don't get how
> > to
> > >> do.
> > >>
> > >> I've conflated two issues here mostly due to needing to get this
> cluster
> > >> stable again.
> > >> - reduce replication
> > >> - remove a partition from a broker, ie. remove the replica
> > >>
> > >> they're very distinct actions, but both would help me in the moment
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, May 21, 2014 at 8:56 AM, Jun Rao <ju...@gmail.com> wrote:
> > >>
> > >>> During the re-assignment, did you move the replica off the old
> broker?
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Jun
> > >>>
> > >>>
> > >>> On Wed, May 21, 2014 at 8:21 AM, David Birdsong <
> > >>> david.birdsong@gmail.com>wrote:
> > >>>
> > >>> > I did that and so now the topic has 4 replicas for a repl count of
> 3,
> > >>> but
> > >>> > only the 'new' replicas exist in the ISR.
> > >>> >
> > >>> > The old broker that I want to clear disk space and generally free
> up
> > >>> > resources has fully synced a topic that I want to disassociate from
> > it.
> > >>> >
> > >>> > Is there a way to do this?
> > >>> >
> > >>> >
> > >>> > On Wed, May 21, 2014 at 7:48 AM, Jun Rao <ju...@gmail.com> wrote:
> > >>> >
> > >>> > > We don't have an exact tool for doing this. You may be able to do
> > >>> that
> > >>> > > through
> > >>> > >
> > >>> > >
> > >>> >
> > >>>
> >
> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby
> > >>> > > specifying fewer replicas.
> > >>> > >
> > >>> > > Thanks,
> > >>> > >
> > >>> > > Jun
> > >>> > >
> > >>> > >
> > >>> > > On Wed, May 21, 2014 at 1:23 AM, David Birdsong <
> > >>> > david.birdsong@gmail.com
> > >>> > > >wrote:
> > >>> > >
> > >>> > > > Is there a way to reduce the replication count? I'm trying to
> > >>> spread
> > >>> > > > existing partitions across more brokers, but it's hard to
> decomm
> > a
> > >>> > > broker.
> > >>> > > > Reducing repl count would suffice for now.
> > >>> > > >
> > >>> > > > Any tips?
> > >>> > > >
> > >>> > > > I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: reduce replication factor
Posted by Jun Rao <ju...@gmail.com>.
Expansion can be done by following
http://kafka.apache.org/documentation.html#basic_ops_cluster_expansion
If you just want to free up a server, you can stop the broker there and
start a broker using the same broker id on a new server. Data should be
automatically replicated to the new server.
Thanks,
Jun
On Wed, May 21, 2014 at 3:23 PM, David Birdsong <da...@gmail.com>wrote:
> Any suggestions? I'm kind of in a bind in that I don't understand how to
> grow the cluster when more capacity is needed--which happens to be right
> now for me.
>
> The only thing I can think that might work is to create new brokers,
> cherry-pick topic/partition pairs and move them, then turn off the old ones
> and forever retire their IDs freeing up my old hardware to come back online
> as a new kafka broker ID.
>
>
> On Wed, May 21, 2014 at 9:16 AM, David Birdsong <david.birdsong@gmail.com
> >wrote:
>
> >
> >
> >
> > On Wed, May 21, 2014 at 9:11 AM, David Birdsong <
> david.birdsong@gmail.com>wrote:
> >
> >> Here's the reassignment json and current "describe" output:
> >> https://gist.github.com/davidbirdsong/32cd0c4f49496a6a32e5
> >>
> >>
> >> In my re-assignment json, I tried to re-assign to 2 when the repl is set
> >> to 3. Once I noticed the the completely new node "133" had appeared in
> the
> >> ISR, I tried stopping 224, wiping kafka completely and then brought 224
> >> back up as 224 again. It promptly replicated the topic, but never joined
> >> the ISR.
> >>
> >
> > to be clear, i don't want it to join the ISR. i'm curious how to make 224
> > forget about the partitions for that topic since i have other plans for
> it.
> >
> >
> >>
> >> How does one move a replica? This is exactly what I'm trying to do.
> >>
> >> My pattern is a common one. I started with a set of 3 kafka brokers. The
> >> load and space is overwhelming them. I'm trying to add new brokers and
> >> spread the partitions to new nodes while removing some of the
> partitions on
> >> the old nodes so as to make room. It's the latter that I don't get how
> to
> >> do.
> >>
> >> I've conflated two issues here mostly due to needing to get this cluster
> >> stable again.
> >> - reduce replication
> >> - remove a partition from a broker, ie. remove the replica
> >>
> >> they're very distinct actions, but both would help me in the moment
> >>
> >>
> >>
> >>
> >> On Wed, May 21, 2014 at 8:56 AM, Jun Rao <ju...@gmail.com> wrote:
> >>
> >>> During the re-assignment, did you move the replica off the old broker?
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>>
> >>> On Wed, May 21, 2014 at 8:21 AM, David Birdsong <
> >>> david.birdsong@gmail.com>wrote:
> >>>
> >>> > I did that and so now the topic has 4 replicas for a repl count of 3,
> >>> but
> >>> > only the 'new' replicas exist in the ISR.
> >>> >
> >>> > The old broker that I want to clear disk space and generally free up
> >>> > resources has fully synced a topic that I want to disassociate from
> it.
> >>> >
> >>> > Is there a way to do this?
> >>> >
> >>> >
> >>> > On Wed, May 21, 2014 at 7:48 AM, Jun Rao <ju...@gmail.com> wrote:
> >>> >
> >>> > > We don't have an exact tool for doing this. You may be able to do
> >>> that
> >>> > > through
> >>> > >
> >>> > >
> >>> >
> >>>
> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby
> >>> > > specifying fewer replicas.
> >>> > >
> >>> > > Thanks,
> >>> > >
> >>> > > Jun
> >>> > >
> >>> > >
> >>> > > On Wed, May 21, 2014 at 1:23 AM, David Birdsong <
> >>> > david.birdsong@gmail.com
> >>> > > >wrote:
> >>> > >
> >>> > > > Is there a way to reduce the replication count? I'm trying to
> >>> spread
> >>> > > > existing partitions across more brokers, but it's hard to decomm
> a
> >>> > > broker.
> >>> > > > Reducing repl count would suffice for now.
> >>> > > >
> >>> > > > Any tips?
> >>> > > >
> >>> > > > I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>
Re: reduce replication factor
Posted by David Birdsong <da...@gmail.com>.
Any suggestions? I'm kind of in a bind in that I don't understand how to
grow the cluster when more capacity is needed--which happens to be right
now for me.
The only thing I can think that might work is to create new brokers,
cherry-pick topic/partition pairs and move them, then turn off the old ones
and forever retire their IDs freeing up my old hardware to come back online
as a new kafka broker ID.
On Wed, May 21, 2014 at 9:16 AM, David Birdsong <da...@gmail.com>wrote:
>
>
>
> On Wed, May 21, 2014 at 9:11 AM, David Birdsong <da...@gmail.com>wrote:
>
>> Here's the reassignment json and current "describe" output:
>> https://gist.github.com/davidbirdsong/32cd0c4f49496a6a32e5
>>
>>
>> In my re-assignment json, I tried to re-assign to 2 when the repl is set
>> to 3. Once I noticed the the completely new node "133" had appeared in the
>> ISR, I tried stopping 224, wiping kafka completely and then brought 224
>> back up as 224 again. It promptly replicated the topic, but never joined
>> the ISR.
>>
>
> to be clear, i don't want it to join the ISR. i'm curious how to make 224
> forget about the partitions for that topic since i have other plans for it.
>
>
>>
>> How does one move a replica? This is exactly what I'm trying to do.
>>
>> My pattern is a common one. I started with a set of 3 kafka brokers. The
>> load and space is overwhelming them. I'm trying to add new brokers and
>> spread the partitions to new nodes while removing some of the partitions on
>> the old nodes so as to make room. It's the latter that I don't get how to
>> do.
>>
>> I've conflated two issues here mostly due to needing to get this cluster
>> stable again.
>> - reduce replication
>> - remove a partition from a broker, ie. remove the replica
>>
>> they're very distinct actions, but both would help me in the moment
>>
>>
>>
>>
>> On Wed, May 21, 2014 at 8:56 AM, Jun Rao <ju...@gmail.com> wrote:
>>
>>> During the re-assignment, did you move the replica off the old broker?
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>>
>>> On Wed, May 21, 2014 at 8:21 AM, David Birdsong <
>>> david.birdsong@gmail.com>wrote:
>>>
>>> > I did that and so now the topic has 4 replicas for a repl count of 3,
>>> but
>>> > only the 'new' replicas exist in the ISR.
>>> >
>>> > The old broker that I want to clear disk space and generally free up
>>> > resources has fully synced a topic that I want to disassociate from it.
>>> >
>>> > Is there a way to do this?
>>> >
>>> >
>>> > On Wed, May 21, 2014 at 7:48 AM, Jun Rao <ju...@gmail.com> wrote:
>>> >
>>> > > We don't have an exact tool for doing this. You may be able to do
>>> that
>>> > > through
>>> > >
>>> > >
>>> >
>>> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby
>>> > > specifying fewer replicas.
>>> > >
>>> > > Thanks,
>>> > >
>>> > > Jun
>>> > >
>>> > >
>>> > > On Wed, May 21, 2014 at 1:23 AM, David Birdsong <
>>> > david.birdsong@gmail.com
>>> > > >wrote:
>>> > >
>>> > > > Is there a way to reduce the replication count? I'm trying to
>>> spread
>>> > > > existing partitions across more brokers, but it's hard to decomm a
>>> > > broker.
>>> > > > Reducing repl count would suffice for now.
>>> > > >
>>> > > > Any tips?
>>> > > >
>>> > > > I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
Re: reduce replication factor
Posted by David Birdsong <da...@gmail.com>.
On Wed, May 21, 2014 at 9:11 AM, David Birdsong <da...@gmail.com>wrote:
> Here's the reassignment json and current "describe" output:
> https://gist.github.com/davidbirdsong/32cd0c4f49496a6a32e5
>
>
> In my re-assignment json, I tried to re-assign to 2 when the repl is set
> to 3. Once I noticed the the completely new node "133" had appeared in the
> ISR, I tried stopping 224, wiping kafka completely and then brought 224
> back up as 224 again. It promptly replicated the topic, but never joined
> the ISR.
>
to be clear, i don't want it to join the ISR. i'm curious how to make 224
forget about the partitions for that topic since i have other plans for it.
>
> How does one move a replica? This is exactly what I'm trying to do.
>
> My pattern is a common one. I started with a set of 3 kafka brokers. The
> load and space is overwhelming them. I'm trying to add new brokers and
> spread the partitions to new nodes while removing some of the partitions on
> the old nodes so as to make room. It's the latter that I don't get how to
> do.
>
> I've conflated two issues here mostly due to needing to get this cluster
> stable again.
> - reduce replication
> - remove a partition from a broker, ie. remove the replica
>
> they're very distinct actions, but both would help me in the moment
>
>
>
>
> On Wed, May 21, 2014 at 8:56 AM, Jun Rao <ju...@gmail.com> wrote:
>
>> During the re-assignment, did you move the replica off the old broker?
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Wed, May 21, 2014 at 8:21 AM, David Birdsong <david.birdsong@gmail.com
>> >wrote:
>>
>> > I did that and so now the topic has 4 replicas for a repl count of 3,
>> but
>> > only the 'new' replicas exist in the ISR.
>> >
>> > The old broker that I want to clear disk space and generally free up
>> > resources has fully synced a topic that I want to disassociate from it.
>> >
>> > Is there a way to do this?
>> >
>> >
>> > On Wed, May 21, 2014 at 7:48 AM, Jun Rao <ju...@gmail.com> wrote:
>> >
>> > > We don't have an exact tool for doing this. You may be able to do that
>> > > through
>> > >
>> > >
>> >
>> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby
>> > > specifying fewer replicas.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > >
>> > > On Wed, May 21, 2014 at 1:23 AM, David Birdsong <
>> > david.birdsong@gmail.com
>> > > >wrote:
>> > >
>> > > > Is there a way to reduce the replication count? I'm trying to spread
>> > > > existing partitions across more brokers, but it's hard to decomm a
>> > > broker.
>> > > > Reducing repl count would suffice for now.
>> > > >
>> > > > Any tips?
>> > > >
>> > > > I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
>> > > >
>> > >
>> >
>>
>
>
Re: reduce replication factor
Posted by David Birdsong <da...@gmail.com>.
Here's the reassignment json and current "describe" output:
https://gist.github.com/davidbirdsong/32cd0c4f49496a6a32e5
In my re-assignment json, I tried to re-assign to 2 when the repl is set to
3. Once I noticed the the completely new node "133" had appeared in the
ISR, I tried stopping 224, wiping kafka completely and then brought 224
back up as 224 again. It promptly replicated the topic, but never joined
the ISR.
How does one move a replica? This is exactly what I'm trying to do.
My pattern is a common one. I started with a set of 3 kafka brokers. The
load and space is overwhelming them. I'm trying to add new brokers and
spread the partitions to new nodes while removing some of the partitions on
the old nodes so as to make room. It's the latter that I don't get how to
do.
I've conflated two issues here mostly due to needing to get this cluster
stable again.
- reduce replication
- remove a partition from a broker, ie. remove the replica
they're very distinct actions, but both would help me in the moment
On Wed, May 21, 2014 at 8:56 AM, Jun Rao <ju...@gmail.com> wrote:
> During the re-assignment, did you move the replica off the old broker?
>
> Thanks,
>
> Jun
>
>
> On Wed, May 21, 2014 at 8:21 AM, David Birdsong <david.birdsong@gmail.com
> >wrote:
>
> > I did that and so now the topic has 4 replicas for a repl count of 3, but
> > only the 'new' replicas exist in the ISR.
> >
> > The old broker that I want to clear disk space and generally free up
> > resources has fully synced a topic that I want to disassociate from it.
> >
> > Is there a way to do this?
> >
> >
> > On Wed, May 21, 2014 at 7:48 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > We don't have an exact tool for doing this. You may be able to do that
> > > through
> > >
> > >
> >
> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby
> > > specifying fewer replicas.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, May 21, 2014 at 1:23 AM, David Birdsong <
> > david.birdsong@gmail.com
> > > >wrote:
> > >
> > > > Is there a way to reduce the replication count? I'm trying to spread
> > > > existing partitions across more brokers, but it's hard to decomm a
> > > broker.
> > > > Reducing repl count would suffice for now.
> > > >
> > > > Any tips?
> > > >
> > > > I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
> > > >
> > >
> >
>
Re: reduce replication factor
Posted by Jun Rao <ju...@gmail.com>.
During the re-assignment, did you move the replica off the old broker?
Thanks,
Jun
On Wed, May 21, 2014 at 8:21 AM, David Birdsong <da...@gmail.com>wrote:
> I did that and so now the topic has 4 replicas for a repl count of 3, but
> only the 'new' replicas exist in the ISR.
>
> The old broker that I want to clear disk space and generally free up
> resources has fully synced a topic that I want to disassociate from it.
>
> Is there a way to do this?
>
>
> On Wed, May 21, 2014 at 7:48 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > We don't have an exact tool for doing this. You may be able to do that
> > through
> >
> >
> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby
> > specifying fewer replicas.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, May 21, 2014 at 1:23 AM, David Birdsong <
> david.birdsong@gmail.com
> > >wrote:
> >
> > > Is there a way to reduce the replication count? I'm trying to spread
> > > existing partitions across more brokers, but it's hard to decomm a
> > broker.
> > > Reducing repl count would suffice for now.
> > >
> > > Any tips?
> > >
> > > I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
> > >
> >
>
Re: reduce replication factor
Posted by David Birdsong <da...@gmail.com>.
I did that and so now the topic has 4 replicas for a repl count of 3, but
only the 'new' replicas exist in the ISR.
The old broker that I want to clear disk space and generally free up
resources has fully synced a topic that I want to disassociate from it.
Is there a way to do this?
On Wed, May 21, 2014 at 7:48 AM, Jun Rao <ju...@gmail.com> wrote:
> We don't have an exact tool for doing this. You may be able to do that
> through
>
> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby
> specifying fewer replicas.
>
> Thanks,
>
> Jun
>
>
> On Wed, May 21, 2014 at 1:23 AM, David Birdsong <david.birdsong@gmail.com
> >wrote:
>
> > Is there a way to reduce the replication count? I'm trying to spread
> > existing partitions across more brokers, but it's hard to decomm a
> broker.
> > Reducing repl count would suffice for now.
> >
> > Any tips?
> >
> > I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
> >
>
Re: reduce replication factor
Posted by Jun Rao <ju...@gmail.com>.
We don't have an exact tool for doing this. You may be able to do that
through
http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby
specifying fewer replicas.
Thanks,
Jun
On Wed, May 21, 2014 at 1:23 AM, David Birdsong <da...@gmail.com>wrote:
> Is there a way to reduce the replication count? I'm trying to spread
> existing partitions across more brokers, but it's hard to decomm a broker.
> Reducing repl count would suffice for now.
>
> Any tips?
>
> I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
>
Re: reduce replication factor
Posted by Todd Palino <tp...@linkedin.com>.
It is possible to do this using the kafka-reassign-partitions admin
command. You can explicitly set the replica list for a partition to be
less than the current replication factor, and that will effectively reduce
it. However, I will say that you should be really sure that you want to do
this. If, for example, you reduce the RF from 2 to 1, you will not be able
to do any maintenance on a live cluster without taking partitions offline.
-Todd
On 5/21/14, 1:23 AM, "David Birdsong" <da...@gmail.com> wrote:
>Is there a way to reduce the replication count? I'm trying to spread
>existing partitions across more brokers, but it's hard to decomm a broker.
>Reducing repl count would suffice for now.
>
>Any tips?
>
>I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
Re: reduce replication factor
Posted by Clark Haskins <ch...@linkedin.com>.
Yes you can. You can use the partition-reassignment tool and move it to a
smaller number of replicas.
-Clark
Clark Elliott Haskins III
LinkedIn DDS Site Reliability Engineer
Kafka, Zookeeper, Samza SRE
Mobile: 505.385.1484
BlueJeans: https://www.bluejeans.com/chaskins
chaskins@linkedin.com
https://www.linkedin.com/in/clarkhaskins
There is no place like 127.0.0.1
On 5/21/14, 1:23 AM, "David Birdsong" <da...@gmail.com> wrote:
>Is there a way to reduce the replication count? I'm trying to spread
>existing partitions across more brokers, but it's hard to decomm a broker.
>Reducing repl count would suffice for now.
>
>Any tips?
>
>I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)