You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Satish Bhatti <ct...@gmail.com> on 2009/05/04 19:35:03 UTC
Moving ZooKeeper Servers
Here is the scenario:
(1) We have 2 live instances of our application, running on 2 different sets
of machines, a preproduction instance and a production instance. As part of
this setup we have 5 ZooKeeper servers running on machines pre-1 thru pre-5.
They have been running for several days.
(2) At some point, we want to switch the preproduction instance to be the
production instance. For the ZooKeeper servers, we will copy the data +
logs directories from the pre machines currently running ZooKeeper to the
prod machines that will be running ZooKeeper, and start up ZooKeeper on
those machines. Is this all that is necessary so that the new ZooKeeper
cluster effectively continues from where the pre cluster left off? Am I
missing something?
Thanks,
Satish
RE: Moving ZooKeeper Servers
Posted by Benjamin Reed <br...@yahoo-inc.com>.
yes, /zookeeper is part of the reserved namespace for zookeeper internals. you should ignore it for such things.
ben
________________________________________
From: Satish Bhatti [cthd2001@gmail.com]
Sent: Wednesday, May 06, 2009 2:57 PM
To: zookeeper-user@hadoop.apache.org
Subject: Re: Moving ZooKeeper Servers
I ended up going with that suggestion, a short recursive function did the
trick! However, I noticed the following nodes:
/zookeeper
/zookeeper/quota
that were not created by me. So I ignored them. Is this correct?
Satish
On Mon, May 4, 2009 at 4:33 PM, Ted Dunning <te...@gmail.com> wrote:
> In fact, the much, much simpler approach of bringing up the production ZK
> cluster and simply writing a program to read from the pre-production
> cluster
> and write to the production one is much more sound. If you can't do that,
> you may need to rethink your processes since they are likely to be delicate
> for other reasons as well.
>
> On Mon, May 4, 2009 at 2:35 PM, Mahadev Konar <ma...@yahoo-inc.com>
> wrote:
>
> > So, zookeeper would work fine if you are careful with above but I would
> > vote
> > against doing this for production since the above is pretty easy to mess
> > up.
> >
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
> 111 West Evelyn Ave. Ste. 202
> Sunnyvale, CA 94086
> www.deepdyve.com
> 858-414-0013 (m)
> 408-773-0220 (fax)
>
Re: Moving ZooKeeper Servers
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Yes, that is correct.
The quota node is a zookeeper service node for quota's in zookeeper.
mahadev
On 5/6/09 2:57 PM, "Satish Bhatti" <ct...@gmail.com> wrote:
> I ended up going with that suggestion, a short recursive function did the
> trick! However, I noticed the following nodes:
> /zookeeper
> /zookeeper/quota
>
> that were not created by me. So I ignored them. Is this correct?
>
> Satish
>
>
> On Mon, May 4, 2009 at 4:33 PM, Ted Dunning <te...@gmail.com> wrote:
>
>> In fact, the much, much simpler approach of bringing up the production ZK
>> cluster and simply writing a program to read from the pre-production
>> cluster
>> and write to the production one is much more sound. If you can't do that,
>> you may need to rethink your processes since they are likely to be delicate
>> for other reasons as well.
>>
>> On Mon, May 4, 2009 at 2:35 PM, Mahadev Konar <ma...@yahoo-inc.com>
>> wrote:
>>
>>> So, zookeeper would work fine if you are careful with above but I would
>>> vote
>>> against doing this for production since the above is pretty easy to mess
>>> up.
>>>
>>
>>
>>
>> --
>> Ted Dunning, CTO
>> DeepDyve
>>
>> 111 West Evelyn Ave. Ste. 202
>> Sunnyvale, CA 94086
>> www.deepdyve.com
>> 858-414-0013 (m)
>> 408-773-0220 (fax)
>>
Re: Moving ZooKeeper Servers
Posted by Satish Bhatti <ct...@gmail.com>.
I ended up going with that suggestion, a short recursive function did the
trick! However, I noticed the following nodes:
/zookeeper
/zookeeper/quota
that were not created by me. So I ignored them. Is this correct?
Satish
On Mon, May 4, 2009 at 4:33 PM, Ted Dunning <te...@gmail.com> wrote:
> In fact, the much, much simpler approach of bringing up the production ZK
> cluster and simply writing a program to read from the pre-production
> cluster
> and write to the production one is much more sound. If you can't do that,
> you may need to rethink your processes since they are likely to be delicate
> for other reasons as well.
>
> On Mon, May 4, 2009 at 2:35 PM, Mahadev Konar <ma...@yahoo-inc.com>
> wrote:
>
> > So, zookeeper would work fine if you are careful with above but I would
> > vote
> > against doing this for production since the above is pretty easy to mess
> > up.
> >
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
> 111 West Evelyn Ave. Ste. 202
> Sunnyvale, CA 94086
> www.deepdyve.com
> 858-414-0013 (m)
> 408-773-0220 (fax)
>
Re: Moving ZooKeeper Servers
Posted by Ted Dunning <te...@gmail.com>.
In fact, the much, much simpler approach of bringing up the production ZK
cluster and simply writing a program to read from the pre-production cluster
and write to the production one is much more sound. If you can't do that,
you may need to rethink your processes since they are likely to be delicate
for other reasons as well.
On Mon, May 4, 2009 at 2:35 PM, Mahadev Konar <ma...@yahoo-inc.com> wrote:
> So, zookeeper would work fine if you are careful with above but I would
> vote
> against doing this for production since the above is pretty easy to mess
> up.
>
--
Ted Dunning, CTO
DeepDyve
111 West Evelyn Ave. Ste. 202
Sunnyvale, CA 94086
www.deepdyve.com
858-414-0013 (m)
408-773-0220 (fax)
Re: Moving ZooKeeper Servers
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Satish,
Is the re generation of state in production something that is not
acceptable? Copying over the whole datadir and datalogdir as it is
maintaining the dirctory structure would be necessary.
Also, in general this is a bad idea (just to warn you) since you would have
to be careful with data copying ( making sure that their is one to one
mapping between the data copying from pre prod to prod)-- meaning
Pre prod1 -> prod1 (copying from pre prod1 to prod 1)
Pre prod2 -> prod2 (copy from pre prod 2 to prod 2).
The one to one mapping is essential to make sure data isnt lost.
Also, you have to make sure htat you have a clean database in prod1 and you
do not have files in production that overlaps old file from production and
the new files you copied over from pre production. This will cause database
corruption since you will have an overlap of database from pre prod and old
production.
So, zookeeper would work fine if you are careful with above but I would vote
against doing this for production since the above is pretty easy to mess up.
mahadev
On 5/4/09 11:10 AM, "Ted Dunning" <te...@gmail.com> wrote:
> I think it would be easier to add the production machines to the cluster one
> by one and then remove the pre-production ZK instances from the cluster one
> by one.
>
> This gives you continuity that you lack otherwise. Adding machines is a
> matter of changing the configuration on each ZK and restarting ZK on that
> machine. You could add the machines in a lump if you don't add so many as
> to prevent the cluster from having a quorum. The configuration change and
> restart can be easily scripted and goes quite quickly.
>
> After the hand-off, you can bring the pre-production machines machines back
> up with a smaller cluster configuration.
>
> Of course, this trick only works if you have no production ZK already in
> place so it won't work the second time around. It is also a bit unusual for
> the complete state of a pre-production staging cluster to be important
> enough to preserve.
>
> On Mon, May 4, 2009 at 10:35 AM, Satish Bhatti <ct...@gmail.com> wrote:
>
>> ... (2) At some point, we want to switch the preproduction instance to be
>> the
>> production instance. For the ZooKeeper servers, we will copy the data +
>> logs directories from the pre machines currently running ZooKeeper to the
>> prod machines that will be running ZooKeeper, and start up ZooKeeper on
>> those machines. Is this all that is necessary so that the new ZooKeeper
>> cluster effectively continues from where the pre cluster left off? Am I
>> missing something?
>>
>> --
> Ted Dunning, CTO
> DeepDyve
Re: Moving ZooKeeper Servers
Posted by Ted Dunning <te...@gmail.com>.
I think it would be easier to add the production machines to the cluster one
by one and then remove the pre-production ZK instances from the cluster one
by one.
This gives you continuity that you lack otherwise. Adding machines is a
matter of changing the configuration on each ZK and restarting ZK on that
machine. You could add the machines in a lump if you don't add so many as
to prevent the cluster from having a quorum. The configuration change and
restart can be easily scripted and goes quite quickly.
After the hand-off, you can bring the pre-production machines machines back
up with a smaller cluster configuration.
Of course, this trick only works if you have no production ZK already in
place so it won't work the second time around. It is also a bit unusual for
the complete state of a pre-production staging cluster to be important
enough to preserve.
On Mon, May 4, 2009 at 10:35 AM, Satish Bhatti <ct...@gmail.com> wrote:
> ... (2) At some point, we want to switch the preproduction instance to be
> the
> production instance. For the ZooKeeper servers, we will copy the data +
> logs directories from the pre machines currently running ZooKeeper to the
> prod machines that will be running ZooKeeper, and start up ZooKeeper on
> those machines. Is this all that is necessary so that the new ZooKeeper
> cluster effectively continues from where the pre cluster left off? Am I
> missing something?
>
> --
Ted Dunning, CTO
DeepDyve