You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Helin Xiang <xk...@gmail.com> on 2015/05/21 15:12:51 UTC
Re: Live Upgrade
Hi, Guozhang
Is there a way to monitor/check if a broker has catched up most of the
replicas of topics in sync?
We are considering an upgrade to 0.8.2.1. A rolling upgrade plan seems
possible because we don't wan't to lose any data. It occured to me that
rolling upgrade wihout control can still lose data.
suppose broker A and broker B both has partition 1.
step 1. broker A did an upgrade. partition 1's leader became broker B, data
will come into broker B.
step 2. broker B did an upgrade too. If broker A has not catched all the
recent data from broker B, after broker A has become leader of partition 1,
the recent data will lose.
the only way seems to wait enough time between step 2 and step 1. but we
don't know how much is enough.
any better solutions for rolling upgrade?
Thanks
On Fri, Oct 17, 2014 at 11:12 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Balaji,
>
> You could do a rolling bounce of the brokers to do the in-place upgrade if
> your partitions have at least two replicas. After that you may probably
> need to rebalance the leaders if they are not balanced any more.
>
> Guozhang
>
> On Wed, Oct 15, 2014 at 10:53 AM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com
> > wrote:
>
> > Hi Guys,
> >
> > Can you guys share any experiences you had with Live Upgrade ?.
> >
> > How reliable is it ?.Did we lose messages ?.
> >
> > What issues did you guys face when doing live upgrade.
> >
> > We are planning to upgrade to 0.8.2 from 0.8-beta before we move our web
> > methods broker based messaging layer to Kafka.
> >
> > We would like to get your inputs.
> >
> > Thanks,
> >
> > Balaji
> >
>
>
>
> --
> -- Guozhang
>
--
*Best Regards向河林*
Re: Live Upgrade
Posted by Helin Xiang <xk...@gmail.com>.
Great! That fits the need.
Thanks, Guozhang.
On Fri, May 22, 2015 at 2:06 AM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Helin,
>
> You can monitor the under_replicated_partitions count. More specifically
> when you shutdown broker A, URP would increase to non-zero values, and when
> you restart A its value will gradually decrease to 0, only bounce broker B
> when URP drops to 0.
>
> Guozhang
>
> On Thu, May 21, 2015 at 6:12 AM, Helin Xiang <xk...@gmail.com> wrote:
>
> > Hi, Guozhang
> >
> > Is there a way to monitor/check if a broker has catched up most of the
> > replicas of topics in sync?
> >
> > We are considering an upgrade to 0.8.2.1. A rolling upgrade plan seems
> > possible because we don't wan't to lose any data. It occured to me that
> > rolling upgrade wihout control can still lose data.
> >
> > suppose broker A and broker B both has partition 1.
> >
> > step 1. broker A did an upgrade. partition 1's leader became broker B,
> data
> > will come into broker B.
> > step 2. broker B did an upgrade too. If broker A has not catched all the
> > recent data from broker B, after broker A has become leader of partition
> 1,
> > the recent data will lose.
> >
> > the only way seems to wait enough time between step 2 and step 1. but we
> > don't know how much is enough.
> >
> > any better solutions for rolling upgrade?
> >
> > Thanks
> >
> >
> > On Fri, Oct 17, 2014 at 11:12 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> >
> > > Hi Balaji,
> > >
> > > You could do a rolling bounce of the brokers to do the in-place upgrade
> > if
> > > your partitions have at least two replicas. After that you may probably
> > > need to rebalance the leaders if they are not balanced any more.
> > >
> > > Guozhang
> > >
> > > On Wed, Oct 15, 2014 at 10:53 AM, Seshadri, Balaji <
> > > Balaji.Seshadri@dish.com
> > > > wrote:
> > >
> > > > Hi Guys,
> > > >
> > > > Can you guys share any experiences you had with Live Upgrade ?.
> > > >
> > > > How reliable is it ?.Did we lose messages ?.
> > > >
> > > > What issues did you guys face when doing live upgrade.
> > > >
> > > > We are planning to upgrade to 0.8.2 from 0.8-beta before we move our
> > web
> > > > methods broker based messaging layer to Kafka.
> > > >
> > > > We would like to get your inputs.
> > > >
> > > > Thanks,
> > > >
> > > > Balaji
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> >
> > --
> >
> >
> > *Best Regards向河林*
> >
>
>
>
> --
> -- Guozhang
>
--
*Best Regards向河林*
Re: Live Upgrade
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Helin,
You can monitor the under_replicated_partitions count. More specifically
when you shutdown broker A, URP would increase to non-zero values, and when
you restart A its value will gradually decrease to 0, only bounce broker B
when URP drops to 0.
Guozhang
On Thu, May 21, 2015 at 6:12 AM, Helin Xiang <xk...@gmail.com> wrote:
> Hi, Guozhang
>
> Is there a way to monitor/check if a broker has catched up most of the
> replicas of topics in sync?
>
> We are considering an upgrade to 0.8.2.1. A rolling upgrade plan seems
> possible because we don't wan't to lose any data. It occured to me that
> rolling upgrade wihout control can still lose data.
>
> suppose broker A and broker B both has partition 1.
>
> step 1. broker A did an upgrade. partition 1's leader became broker B, data
> will come into broker B.
> step 2. broker B did an upgrade too. If broker A has not catched all the
> recent data from broker B, after broker A has become leader of partition 1,
> the recent data will lose.
>
> the only way seems to wait enough time between step 2 and step 1. but we
> don't know how much is enough.
>
> any better solutions for rolling upgrade?
>
> Thanks
>
>
> On Fri, Oct 17, 2014 at 11:12 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
>
> > Hi Balaji,
> >
> > You could do a rolling bounce of the brokers to do the in-place upgrade
> if
> > your partitions have at least two replicas. After that you may probably
> > need to rebalance the leaders if they are not balanced any more.
> >
> > Guozhang
> >
> > On Wed, Oct 15, 2014 at 10:53 AM, Seshadri, Balaji <
> > Balaji.Seshadri@dish.com
> > > wrote:
> >
> > > Hi Guys,
> > >
> > > Can you guys share any experiences you had with Live Upgrade ?.
> > >
> > > How reliable is it ?.Did we lose messages ?.
> > >
> > > What issues did you guys face when doing live upgrade.
> > >
> > > We are planning to upgrade to 0.8.2 from 0.8-beta before we move our
> web
> > > methods broker based messaging layer to Kafka.
> > >
> > > We would like to get your inputs.
> > >
> > > Thanks,
> > >
> > > Balaji
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
>
>
> *Best Regards向河林*
>
--
-- Guozhang