You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by 王国栋 <wa...@gmail.com> on 2013/02/17 11:19:53 UTC
Re: Graceful termination of kafka broker after draining all the data consumed
Hi Jun,
If we use high level producer based on zookeeper, how can we decommission a
broker without message loss?
Since we want to partition the log with IP, if all the brokers use the
same vip, we can not use the customized partition strategy.
Thanks.
On Mon, Jan 7, 2013 at 12:52 AM, Jun Rao <ju...@gmail.com> wrote:
> In 0.7, one way to do this is to use a vip. All producers send data to the
> vip. To decommission a broker, you first take the broker out of vip so no
> new data will be produced to it. Then you let the consumer drain the data
> (you can use ConsumerOffsetChecker to check if all data has been consumed).
> Finally, you can shut down the broker.
>
> This will be much easier in 0.8 because of replication.
>
> Thanks,
>
> Jun
>
> On Sat, Jan 5, 2013 at 11:34 PM, Bae, Jae Hyeon <me...@gmail.com>
> wrote:
>
> > Hi
> >
> > If I want to terminate kafka broker gracefully. Before termination, it
> > should stop receiving the traffic from producers and wait until all
> > data will be consumed.
> >
> > I don't think that kafka 0.7.x is supporting this feature. If I want
> > to implement this feature for myself, could you give me a brief sketch
> > of implementation?
> >
> > Thank you
> > Best, Jae
> >
>
--
Guodong Wang
王国栋
Re: Graceful termination of kafka broker after draining all the data consumed
Posted by 王国栋 <wa...@gmail.com>.
Thanks Jun. I will check more info about 0.8
On Tue, Feb 19, 2013 at 12:51 AM, Jun Rao <ju...@gmail.com> wrote:
> In 0.7, it's not very easy to decommission a broker using ZK based
> producers. It's possible to do that with a vip (then you can't do
> partitioning). In 0.8 (probably 0.8.1), you can use a tool to move all
> partitions off a broker first and then decommission it.
>
> Thanks,
>
> Jun
>
>
> On Sun, Feb 17, 2013 at 2:19 AM, 王国栋 <wa...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > If we use high level producer based on zookeeper, how can we
> decommission a
> > broker without message loss?
> >
> > Since we want to partition the log with IP, if all the brokers use the
> > same vip, we can not use the customized partition strategy.
> >
> > Thanks.
> >
> >
> > On Mon, Jan 7, 2013 at 12:52 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > In 0.7, one way to do this is to use a vip. All producers send data to
> > the
> > > vip. To decommission a broker, you first take the broker out of vip so
> > no
> > > new data will be produced to it. Then you let the consumer drain the
> data
> > > (you can use ConsumerOffsetChecker to check if all data has been
> > consumed).
> > > Finally, you can shut down the broker.
> > >
> > > This will be much easier in 0.8 because of replication.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sat, Jan 5, 2013 at 11:34 PM, Bae, Jae Hyeon <me...@gmail.com>
> > > wrote:
> > >
> > > > Hi
> > > >
> > > > If I want to terminate kafka broker gracefully. Before termination,
> it
> > > > should stop receiving the traffic from producers and wait until all
> > > > data will be consumed.
> > > >
> > > > I don't think that kafka 0.7.x is supporting this feature. If I want
> > > > to implement this feature for myself, could you give me a brief
> sketch
> > > > of implementation?
> > > >
> > > > Thank you
> > > > Best, Jae
> > > >
> > >
> >
> >
> >
> > --
> > Guodong Wang
> > 王国栋
> >
>
--
Guodong Wang
王国栋
Re: Graceful termination of kafka broker after draining all the data consumed
Posted by Jun Rao <ju...@gmail.com>.
In 0.7, it's not very easy to decommission a broker using ZK based
producers. It's possible to do that with a vip (then you can't do
partitioning). In 0.8 (probably 0.8.1), you can use a tool to move all
partitions off a broker first and then decommission it.
Thanks,
Jun
On Sun, Feb 17, 2013 at 2:19 AM, 王国栋 <wa...@gmail.com> wrote:
> Hi Jun,
>
> If we use high level producer based on zookeeper, how can we decommission a
> broker without message loss?
>
> Since we want to partition the log with IP, if all the brokers use the
> same vip, we can not use the customized partition strategy.
>
> Thanks.
>
>
> On Mon, Jan 7, 2013 at 12:52 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > In 0.7, one way to do this is to use a vip. All producers send data to
> the
> > vip. To decommission a broker, you first take the broker out of vip so
> no
> > new data will be produced to it. Then you let the consumer drain the data
> > (you can use ConsumerOffsetChecker to check if all data has been
> consumed).
> > Finally, you can shut down the broker.
> >
> > This will be much easier in 0.8 because of replication.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sat, Jan 5, 2013 at 11:34 PM, Bae, Jae Hyeon <me...@gmail.com>
> > wrote:
> >
> > > Hi
> > >
> > > If I want to terminate kafka broker gracefully. Before termination, it
> > > should stop receiving the traffic from producers and wait until all
> > > data will be consumed.
> > >
> > > I don't think that kafka 0.7.x is supporting this feature. If I want
> > > to implement this feature for myself, could you give me a brief sketch
> > > of implementation?
> > >
> > > Thank you
> > > Best, Jae
> > >
> >
>
>
>
> --
> Guodong Wang
> 王国栋
>