You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bookkeeper.apache.org by John Nagro <jn...@hubspot.com> on 2012/05/03 15:39:34 UTC
rolling restarts
hello -
I have a cluster with 5 bk nodes, my client settings are to use 3 at a time
with a quorum of 2. I want to upgrade the software of a live cluster from
an older 4.1-SNAPSHOT to a newer one. I am under the impression that I
should not need to upgrade the filesystem since I've already done that when
moving to 4.1.
Am I correct in that I should be able to bring down the nodes one at a
time, replace the jar, and bring them back up. As long as I do it one at a
time the cluster should continue to operate properly?
also - there doesn't appear to be any sort of graceful shutdown code, and
the docs show ^C in a few places - so thats how i've been bringing nodes
down. that is still the standard procedure?
thanks!
-John Nagro
Re: rolling restarts
Posted by John Nagro <jn...@hubspot.com>.
Thanks Ivan!
On Thu, May 3, 2012 at 11:20 AM, Ivan Kelly <iv...@apache.org> wrote:
> On Thu, May 03, 2012 at 10:45:51AM -0400, John Nagro wrote:
> > Okay - and doing the upgrade to a live cluster? I can do that to 1
> machine
> > at a time while the cluster is still online and everything should operate
> > as normal? I have 5 machines, use 3 at a time, quorum of 2.
> In terms of rolling restarts, if ensemble size if 3 and you have 5
> machines, you can bring down one machine at a time and it'll be
> fine. Any clients writing to that machine will move very quickly to
> using one of the other machines (it'll see the network drop, and
> rebuild an ensemble from the other machines).
>
> Regarding whether you need to run the upgrade code, it depends on what
> the old version of bookkeeper you are using is. If you need to run
> upgrade, the bookie will warn you and fail to start.
>
> https://issues.apache.org/jira/browse/BOOKKEEPER-163 is actually the
> change that requires the upgrade framework.
>
> -Ivan
>
Re: rolling restarts
Posted by Ivan Kelly <iv...@apache.org>.
On Thu, May 03, 2012 at 10:45:51AM -0400, John Nagro wrote:
> Okay - and doing the upgrade to a live cluster? I can do that to 1 machine
> at a time while the cluster is still online and everything should operate
> as normal? I have 5 machines, use 3 at a time, quorum of 2.
In terms of rolling restarts, if ensemble size if 3 and you have 5
machines, you can bring down one machine at a time and it'll be
fine. Any clients writing to that machine will move very quickly to
using one of the other machines (it'll see the network drop, and
rebuild an ensemble from the other machines).
Regarding whether you need to run the upgrade code, it depends on what
the old version of bookkeeper you are using is. If you need to run
upgrade, the bookie will warn you and fail to start.
https://issues.apache.org/jira/browse/BOOKKEEPER-163 is actually the
change that requires the upgrade framework.
-Ivan
Re: rolling restarts
Posted by John Nagro <jn...@hubspot.com>.
Okay - and doing the upgrade to a live cluster? I can do that to 1 machine
at a time while the cluster is still online and everything should operate
as normal? I have 5 machines, use 3 at a time, quorum of 2.
Thanks.
-John
On Thu, May 3, 2012 at 10:36 AM, Uma Maheswara Rao G
<ma...@huawei.com>wrote:
> BOOKKEEPER-172 <https://issues.apache.org/jira/browse/BOOKKEEPER-172>Provide the implementation for upgrades between versions.
>
>
>
> Here fileSystem layout will refer the bookie data layout format which is
> persisted on the directories or ZKs. The new version of software(BK server)
> should be able to load them to memory correctly.
>
>
>
> So, It is better to do the upgrades as suggested in doc. That will take
> care of upgrading your software and also can finalyze if every thing ok. If
> there are any issues, it will roll back as well.
>
>
>
> Let Ivan/Flavio conform about the layout changes between versions and if
> i am wrong BK case.
>
>
>
> Thanks & Regards,
>
> Uma
>
>
> ------------------------------
> *From:* John Nagro [jnagro@hubspot.com]
> *Sent:* Thursday, May 03, 2012 7:51 PM
> *To:* bookkeeper-user@zookeeper.apache.org
> *Subject:* Re: rolling restarts
>
> Thanks Uma - what I actually need to know is if its okay doing the
> restarts on a live cluster. I think my filesystem is okay at this point,
> but I want to upgrade the server software on a live, running cluster. Is it
> fine to ^C, upgrade, start again - as long as I only do it to 1 of the 5
> nodes at a time?
>
> Thanks!
>
> -John
>
> On Thu, May 3, 2012 at 10:10 AM, Uma Maheswara Rao G <maheswara@huawei.com
> > wrote:
>
>> Hi John,
>>
>>
>>
>> Seems Ivan already worked on upgradation framework. I am not sure about
>> the filesystem layout changes with previous version as i was not aware of
>> older versions code changes.
>>
>>
>>
>> Pls Refer
>> BOOKKEEPER-172 <https://issues.apache.org/jira/browse/BOOKKEEPER-172>
>>
>>
>>
>> You can find some details here:
>> http://zookeeper.apache.org/bookkeeper/docs/trunk/bookkeeperConfig.html
>>
>>
>>
>> Hope this will help you.
>>
>>
>>
>>
>>
>> Regards,
>>
>> Uma
>> ------------------------------
>> *From:* John Nagro [jnagro@hubspot.com]
>> *Sent:* Thursday, May 03, 2012 7:09 PM
>> *To:* bookkeeper-user@zookeeper.apache.org
>> *Subject:* rolling restarts
>>
>> hello -
>>
>> I have a cluster with 5 bk nodes, my client settings are to use 3 at a
>> time with a quorum of 2. I want to upgrade the software of a live cluster
>> from an older 4.1-SNAPSHOT to a newer one. I am under the impression that I
>> should not need to upgrade the filesystem since I've already done that when
>> moving to 4.1.
>>
>> Am I correct in that I should be able to bring down the nodes one at a
>> time, replace the jar, and bring them back up. As long as I do it one at a
>> time the cluster should continue to operate properly?
>>
>> also - there doesn't appear to be any sort of graceful shutdown code,
>> and the docs show ^C in a few places - so thats how i've been bringing
>> nodes down. that is still the standard procedure?
>>
>> thanks!
>>
>> -John Nagro
>>
>>
>
RE: rolling restarts
Posted by Uma Maheswara Rao G <ma...@huawei.com>.
BOOKKEEPER-172<https://issues.apache.org/jira/browse/BOOKKEEPER-172> Provide the implementation for upgrades between versions.
Here fileSystem layout will refer the bookie data layout format which is persisted on the directories or ZKs. The new version of software(BK server) should be able to load them to memory correctly.
So, It is better to do the upgrades as suggested in doc. That will take care of upgrading your software and also can finalyze if every thing ok. If there are any issues, it will roll back as well.
Let Ivan/Flavio conform about the layout changes between versions and if i am wrong BK case.
Thanks & Regards,
Uma
________________________________
From: John Nagro [jnagro@hubspot.com]
Sent: Thursday, May 03, 2012 7:51 PM
To: bookkeeper-user@zookeeper.apache.org
Subject: Re: rolling restarts
Thanks Uma - what I actually need to know is if its okay doing the restarts on a live cluster. I think my filesystem is okay at this point, but I want to upgrade the server software on a live, running cluster. Is it fine to ^C, upgrade, start again - as long as I only do it to 1 of the 5 nodes at a time?
Thanks!
-John
On Thu, May 3, 2012 at 10:10 AM, Uma Maheswara Rao G <ma...@huawei.com>> wrote:
Hi John,
Seems Ivan already worked on upgradation framework. I am not sure about the filesystem layout changes with previous version as i was not aware of older versions code changes.
Pls Refer
* BOOKKEEPER-172<https://issues.apache.org/jira/browse/BOOKKEEPER-172>
You can find some details here: http://zookeeper.apache.org/bookkeeper/docs/trunk/bookkeeperConfig.html
Hope this will help you.
Regards,
Uma
________________________________
From: John Nagro [jnagro@hubspot.com<ma...@hubspot.com>]
Sent: Thursday, May 03, 2012 7:09 PM
To: bookkeeper-user@zookeeper.apache.org<ma...@zookeeper.apache.org>
Subject: rolling restarts
hello -
I have a cluster with 5 bk nodes, my client settings are to use 3 at a time with a quorum of 2. I want to upgrade the software of a live cluster from an older 4.1-SNAPSHOT to a newer one. I am under the impression that I should not need to upgrade the filesystem since I've already done that when moving to 4.1.
Am I correct in that I should be able to bring down the nodes one at a time, replace the jar, and bring them back up. As long as I do it one at a time the cluster should continue to operate properly?
also - there doesn't appear to be any sort of graceful shutdown code, and the docs show ^C in a few places - so thats how i've been bringing nodes down. that is still the standard procedure?
thanks!
-John Nagro
Re: rolling restarts
Posted by John Nagro <jn...@hubspot.com>.
Thanks Uma - what I actually need to know is if its okay doing the restarts
on a live cluster. I think my filesystem is okay at this point, but I want
to upgrade the server software on a live, running cluster. Is it fine to
^C, upgrade, start again - as long as I only do it to 1 of the 5 nodes at a
time?
Thanks!
-John
On Thu, May 3, 2012 at 10:10 AM, Uma Maheswara Rao G
<ma...@huawei.com>wrote:
> Hi John,
>
>
>
> Seems Ivan already worked on upgradation framework. I am not sure about
> the filesystem layout changes with previous version as i was not aware of
> older versions code changes.
>
>
>
> Pls Refer
> BOOKKEEPER-172 <https://issues.apache.org/jira/browse/BOOKKEEPER-172>
>
>
>
> You can find some details here:
> http://zookeeper.apache.org/bookkeeper/docs/trunk/bookkeeperConfig.html
>
>
>
> Hope this will help you.
>
>
>
>
>
> Regards,
>
> Uma
> ------------------------------
> *From:* John Nagro [jnagro@hubspot.com]
> *Sent:* Thursday, May 03, 2012 7:09 PM
> *To:* bookkeeper-user@zookeeper.apache.org
> *Subject:* rolling restarts
>
> hello -
>
> I have a cluster with 5 bk nodes, my client settings are to use 3 at a
> time with a quorum of 2. I want to upgrade the software of a live cluster
> from an older 4.1-SNAPSHOT to a newer one. I am under the impression that I
> should not need to upgrade the filesystem since I've already done that when
> moving to 4.1.
>
> Am I correct in that I should be able to bring down the nodes one at a
> time, replace the jar, and bring them back up. As long as I do it one at a
> time the cluster should continue to operate properly?
>
> also - there doesn't appear to be any sort of graceful shutdown code,
> and the docs show ^C in a few places - so thats how i've been bringing
> nodes down. that is still the standard procedure?
>
> thanks!
>
> -John Nagro
>
>
RE: rolling restarts
Posted by Uma Maheswara Rao G <ma...@huawei.com>.
Hi John,
Seems Ivan already worked on upgradation framework. I am not sure about the filesystem layout changes with previous version as i was not aware of older versions code changes.
Pls Refer
* BOOKKEEPER-172<https://issues.apache.org/jira/browse/BOOKKEEPER-172>
You can find some details here: http://zookeeper.apache.org/bookkeeper/docs/trunk/bookkeeperConfig.html
Hope this will help you.
Regards,
Uma
________________________________
From: John Nagro [jnagro@hubspot.com]
Sent: Thursday, May 03, 2012 7:09 PM
To: bookkeeper-user@zookeeper.apache.org
Subject: rolling restarts
hello -
I have a cluster with 5 bk nodes, my client settings are to use 3 at a time with a quorum of 2. I want to upgrade the software of a live cluster from an older 4.1-SNAPSHOT to a newer one. I am under the impression that I should not need to upgrade the filesystem since I've already done that when moving to 4.1.
Am I correct in that I should be able to bring down the nodes one at a time, replace the jar, and bring them back up. As long as I do it one at a time the cluster should continue to operate properly?
also - there doesn't appear to be any sort of graceful shutdown code, and the docs show ^C in a few places - so thats how i've been bringing nodes down. that is still the standard procedure?
thanks!
-John Nagro