You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Sergey Maslyakov <ev...@gmail.com> on 2013/07/05 05:38:40 UTC
Backup/restore of an emsemble
A while ago, Jack Ma asked this question:
http://mail-archives.apache.org/mod_mbox/zookeeper-user/201306.mbox/%3CCAB%2BcfdyPDpbUh5FyDT%3D9mU%3DFCHEA1AZpkF6X0nN1t4mjwqu2tA%40mail.gmail.com%3E
I wonder if there were any helpful suggestions that did not go into the
mailing list.
I am mostly concerned about restoring data in a Zookeeper ensemble.
There is no document at the project web-site that would explain the restore
procedure for a distributed deployment. The home-grown solution that
involves stopping the whole cluster, wiping out databases on all but one
server, restoring the database on one server, and then bring up the cluster
and pray that the populated server becomes the leader and populates the
cluster. Such solution seems to be too error-prone.
Does anyone have recommendations on how to make it robust?
Maybe there is a way to force-populate the ensemble remotely?
Thanks,
/Sergey
Re: Backup/restore of an emsemble
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Exhibitor does not handle snapshots. The backup feature in Exhibitor is for transaction logs only and then only for selected ZNodes in the logs. Exhibitor will create a Lucene index out of the backed up logs so that one can replay any event. IMO - 99% of nodes in ZooKeeper are usually ephemeral and should not be restored.
-JZ
On Jul 9, 2013, at 2:16 AM, Flavio Junqueira <fp...@yahoo.com> wrote:
> Hi Jordan,
>
> I have one question about the Backup/Restore feature of Exhibitor. Perhaps I should ask this question on the project list, but given this recent thread I thought it would be a good idea to ask it here.
>
> This restore doc page:
>
> https://github.com/Netflix/exhibitor/wiki/Restore-UI
>
> refers to logs but not snapshots. What does Exhibitor do about snapshots if anything at all?
>
> -Flavio
>
>
> On Jul 5, 2013, at 11:41 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>
>> I'm curious why Exhibitor doesn't meet your needs.
>>
>> -JZ
>>
>> On Jul 5, 2013, at 9:12 AM, Sergey Maslyakov <ev...@gmail.com> wrote:
>>
>>> Yes, Flavio, I looked at Exhibitor, but I need a pretty granular control
>>> over a cluster of ZK servers. This is why I'm inclined to build something
>>> by hand. So far, a pair of external export and import clients seems like a
>>> promising approach.
>>>
>>> Export would connect to the ensemble and dump out the data into a file on
>>> disk. Import would connect, wipe out the namespace, and then reload the
>>> data from the file that was earlier created by the export client.
>>>
>>>
>>>
>>> On Fri, Jul 5, 2013 at 8:31 AM, Flavio Junqueira <fp...@yahoo.com>wrote:
>>>
>>>> Sergey,
>>>>
>>>> Have you had a look at Exhibitor?
>>>>
>>>> https://github.com/Netflix/exhibitor
>>>>
>>>> -Flavio
>>>>
>>>> -----Original Message-----
>>>> From: Sergey Maslyakov [mailto:evolvah@gmail.com]
>>>> Sent: 05 July 2013 04:39
>>>> To: user@zookeeper.apache.org
>>>> Subject: Backup/restore of an emsemble
>>>>
>>>> A while ago, Jack Ma asked this question:
>>>>
>>>> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201306.mbox/%3CCAB%2
>>>> BcfdyPDpbUh5FyDT%3D9mU%3DFCHEA1AZpkF6X0nN1t4mjwqu2tA%40mail.gmail.com%3E
>>>>
>>>> I wonder if there were any helpful suggestions that did not go into the
>>>> mailing list.
>>>>
>>>> I am mostly concerned about restoring data in a Zookeeper ensemble.
>>>>
>>>> There is no document at the project web-site that would explain the restore
>>>> procedure for a distributed deployment. The home-grown solution that
>>>> involves stopping the whole cluster, wiping out databases on all but one
>>>> server, restoring the database on one server, and then bring up the cluster
>>>> and pray that the populated server becomes the leader and populates the
>>>> cluster. Such solution seems to be too error-prone.
>>>>
>>>> Does anyone have recommendations on how to make it robust?
>>>>
>>>> Maybe there is a way to force-populate the ensemble remotely?
>>>>
>>>>
>>>> Thanks,
>>>> /Sergey
>>>>
>>>>
>>
>
Re: Backup/restore of an emsemble
Posted by Flavio Junqueira <fp...@yahoo.com>.
Hi Jordan,
I have one question about the Backup/Restore feature of Exhibitor. Perhaps I should ask this question on the project list, but given this recent thread I thought it would be a good idea to ask it here.
This restore doc page:
https://github.com/Netflix/exhibitor/wiki/Restore-UI
refers to logs but not snapshots. What does Exhibitor do about snapshots if anything at all?
-Flavio
On Jul 5, 2013, at 11:41 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
> I'm curious why Exhibitor doesn't meet your needs.
>
> -JZ
>
> On Jul 5, 2013, at 9:12 AM, Sergey Maslyakov <ev...@gmail.com> wrote:
>
>> Yes, Flavio, I looked at Exhibitor, but I need a pretty granular control
>> over a cluster of ZK servers. This is why I'm inclined to build something
>> by hand. So far, a pair of external export and import clients seems like a
>> promising approach.
>>
>> Export would connect to the ensemble and dump out the data into a file on
>> disk. Import would connect, wipe out the namespace, and then reload the
>> data from the file that was earlier created by the export client.
>>
>>
>>
>> On Fri, Jul 5, 2013 at 8:31 AM, Flavio Junqueira <fp...@yahoo.com>wrote:
>>
>>> Sergey,
>>>
>>> Have you had a look at Exhibitor?
>>>
>>> https://github.com/Netflix/exhibitor
>>>
>>> -Flavio
>>>
>>> -----Original Message-----
>>> From: Sergey Maslyakov [mailto:evolvah@gmail.com]
>>> Sent: 05 July 2013 04:39
>>> To: user@zookeeper.apache.org
>>> Subject: Backup/restore of an emsemble
>>>
>>> A while ago, Jack Ma asked this question:
>>>
>>> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201306.mbox/%3CCAB%2
>>> BcfdyPDpbUh5FyDT%3D9mU%3DFCHEA1AZpkF6X0nN1t4mjwqu2tA%40mail.gmail.com%3E
>>>
>>> I wonder if there were any helpful suggestions that did not go into the
>>> mailing list.
>>>
>>> I am mostly concerned about restoring data in a Zookeeper ensemble.
>>>
>>> There is no document at the project web-site that would explain the restore
>>> procedure for a distributed deployment. The home-grown solution that
>>> involves stopping the whole cluster, wiping out databases on all but one
>>> server, restoring the database on one server, and then bring up the cluster
>>> and pray that the populated server becomes the leader and populates the
>>> cluster. Such solution seems to be too error-prone.
>>>
>>> Does anyone have recommendations on how to make it robust?
>>>
>>> Maybe there is a way to force-populate the ensemble remotely?
>>>
>>>
>>> Thanks,
>>> /Sergey
>>>
>>>
>
Re: Backup/restore of an emsemble
Posted by Sergey Maslyakov <ev...@gmail.com>.
Jordan,
The existing system is pretty rigid and it is built without Exhibitor.
Introducing it into the picture will dramatically increase the amount of
work around deployment and software verification. This is not something I'm
ready to commit to at this moment in time.
But maybe I don't know something about Exhibitor that would increase my
comfort level?
In the meantime, I posted this in the "dev" mailing list:
http://mail-archives.apache.org/mod_mbox/zookeeper-dev/201307.mbox/%3CCAB3mbkSXy9V4F9_URQ1_YP4rMU1OEAKwWMi-JAzzNrA%2B4BdJPw%40mail.gmail.com%3E
On Fri, Jul 5, 2013 at 4:41 PM, Jordan Zimmerman <jordan@jordanzimmerman.com
> wrote:
> I'm curious why Exhibitor doesn't meet your needs.
>
> -JZ
>
> On Jul 5, 2013, at 9:12 AM, Sergey Maslyakov <ev...@gmail.com> wrote:
>
> > Yes, Flavio, I looked at Exhibitor, but I need a pretty granular control
> > over a cluster of ZK servers. This is why I'm inclined to build something
> > by hand. So far, a pair of external export and import clients seems like
> a
> > promising approach.
> >
> > Export would connect to the ensemble and dump out the data into a file on
> > disk. Import would connect, wipe out the namespace, and then reload the
> > data from the file that was earlier created by the export client.
> >
> >
> >
> > On Fri, Jul 5, 2013 at 8:31 AM, Flavio Junqueira <fpjunqueira@yahoo.com
> >wrote:
> >
> >> Sergey,
> >>
> >> Have you had a look at Exhibitor?
> >>
> >> https://github.com/Netflix/exhibitor
> >>
> >> -Flavio
> >>
> >> -----Original Message-----
> >> From: Sergey Maslyakov [mailto:evolvah@gmail.com]
> >> Sent: 05 July 2013 04:39
> >> To: user@zookeeper.apache.org
> >> Subject: Backup/restore of an emsemble
> >>
> >> A while ago, Jack Ma asked this question:
> >>
> >>
> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201306.mbox/%3CCAB%2
> >> BcfdyPDpbUh5FyDT%3D9mU%3DFCHEA1AZpkF6X0nN1t4mjwqu2tA%40mail.gmail.com
> %3E
> >>
> >> I wonder if there were any helpful suggestions that did not go into the
> >> mailing list.
> >>
> >> I am mostly concerned about restoring data in a Zookeeper ensemble.
> >>
> >> There is no document at the project web-site that would explain the
> restore
> >> procedure for a distributed deployment. The home-grown solution that
> >> involves stopping the whole cluster, wiping out databases on all but one
> >> server, restoring the database on one server, and then bring up the
> cluster
> >> and pray that the populated server becomes the leader and populates the
> >> cluster. Such solution seems to be too error-prone.
> >>
> >> Does anyone have recommendations on how to make it robust?
> >>
> >> Maybe there is a way to force-populate the ensemble remotely?
> >>
> >>
> >> Thanks,
> >> /Sergey
> >>
> >>
>
>
Re: Backup/restore of an emsemble
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
I'm curious why Exhibitor doesn't meet your needs.
-JZ
On Jul 5, 2013, at 9:12 AM, Sergey Maslyakov <ev...@gmail.com> wrote:
> Yes, Flavio, I looked at Exhibitor, but I need a pretty granular control
> over a cluster of ZK servers. This is why I'm inclined to build something
> by hand. So far, a pair of external export and import clients seems like a
> promising approach.
>
> Export would connect to the ensemble and dump out the data into a file on
> disk. Import would connect, wipe out the namespace, and then reload the
> data from the file that was earlier created by the export client.
>
>
>
> On Fri, Jul 5, 2013 at 8:31 AM, Flavio Junqueira <fp...@yahoo.com>wrote:
>
>> Sergey,
>>
>> Have you had a look at Exhibitor?
>>
>> https://github.com/Netflix/exhibitor
>>
>> -Flavio
>>
>> -----Original Message-----
>> From: Sergey Maslyakov [mailto:evolvah@gmail.com]
>> Sent: 05 July 2013 04:39
>> To: user@zookeeper.apache.org
>> Subject: Backup/restore of an emsemble
>>
>> A while ago, Jack Ma asked this question:
>>
>> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201306.mbox/%3CCAB%2
>> BcfdyPDpbUh5FyDT%3D9mU%3DFCHEA1AZpkF6X0nN1t4mjwqu2tA%40mail.gmail.com%3E
>>
>> I wonder if there were any helpful suggestions that did not go into the
>> mailing list.
>>
>> I am mostly concerned about restoring data in a Zookeeper ensemble.
>>
>> There is no document at the project web-site that would explain the restore
>> procedure for a distributed deployment. The home-grown solution that
>> involves stopping the whole cluster, wiping out databases on all but one
>> server, restoring the database on one server, and then bring up the cluster
>> and pray that the populated server becomes the leader and populates the
>> cluster. Such solution seems to be too error-prone.
>>
>> Does anyone have recommendations on how to make it robust?
>>
>> Maybe there is a way to force-populate the ensemble remotely?
>>
>>
>> Thanks,
>> /Sergey
>>
>>
Re: Backup/restore of an emsemble
Posted by Edward Ribeiro <ed...@gmail.com>.
Yes! :-) Please, cc to me.
Edward
Em 05/07/2013 17:53, "Sergey Maslyakov" <ev...@gmail.com> escreveu:
> Thank you for the reference, Edward! It looks like Guano mostly
> accomplished what I want out of it. I spent a little time today digging in
> the guts of ZK and found a few ways to make taking a backup more efficient
> by delegating all the work to the server. Description of these ways does
> not fit the boundaries of a user mailing list and I intend to send a little
> write-up into the "dev" mailing list. Please let me know if you are
> interested in the subject and I can CC you.
>
>
> On Fri, Jul 5, 2013 at 12:43 PM, Edward Ribeiro <edward.ribeiro@gmail.com
> >wrote:
>
> > Sergey,
> >
> > Have you looked into the Guano Project (https://github.com/d2fn/guano) ?
> > It
> > seems to do exactly what you want, or at least as close a dump/restore so
> > that you can start from there.
> >
> > Edward
> >
> >
> > On Fri, Jul 5, 2013 at 1:40 PM, Sergey Maslyakov <ev...@gmail.com>
> > wrote:
> >
> > > The nature of the system that I am working with is mostly read-heavy.
> > > Writes happen rarely. In he event of disaster recovery to some point in
> > > time in the past, the system should sustain minor inconsistencies.
> > >
> > > I am also considering to have Zookeeper server create a copy of the
> > > DataTree, and then serialize it into a file, which will later be picked
> > up
> > > by the import client. This could be the most efficient way of taking a
> > > backup.
> > >
> > >
> > >
> > > On Fri, Jul 5, 2013 at 11:33 AM, Flavio Junqueira <
> fpjunqueira@yahoo.com
> > > >wrote:
> > >
> > > > In your approach, would you "lock" the zookeeper state and read the
> > data
> > > > tree using getData/getChildren? If you have concurrent updates, then
> > you
> > > > may
> > > > end up having an inconsistent snapshot.
> > > >
> > > > -Flavio
> > > >
> > > > -----Original Message-----
> > > > From: Sergey Maslyakov [mailto:evolvah@gmail.com]
> > > > Sent: 05 July 2013 17:12
> > > > To: user@zookeeper.apache.org
> > > > Subject: Re: Backup/restore of an emsemble
> > > >
> > > > Yes, Flavio, I looked at Exhibitor, but I need a pretty granular
> > control
> > > > over a cluster of ZK servers. This is why I'm inclined to build
> > something
> > > > by
> > > > hand. So far, a pair of external export and import clients seems
> like a
> > > > promising approach.
> > > >
> > > > Export would connect to the ensemble and dump out the data into a
> file
> > on
> > > > disk. Import would connect, wipe out the namespace, and then reload
> the
> > > > data
> > > > from the file that was earlier created by the export client.
> > > >
> > > >
> > > >
> > > > On Fri, Jul 5, 2013 at 8:31 AM, Flavio Junqueira
> > > > <fp...@yahoo.com>wrote:
> > > >
> > > > > Sergey,
> > > > >
> > > > > Have you had a look at Exhibitor?
> > > > >
> > > > > https://github.com/Netflix/exhibitor
> > > > >
> > > > > -Flavio
> > > > >
> > > > > -----Original Message-----
> > > > > From: Sergey Maslyakov [mailto:evolvah@gmail.com]
> > > > > Sent: 05 July 2013 04:39
> > > > > To: user@zookeeper.apache.org
> > > > > Subject: Backup/restore of an emsemble
> > > > >
> > > > > A while ago, Jack Ma asked this question:
> > > > >
> > > > >
> > http://mail-archives.apache.org/mod_mbox/zookeeper-user/201306.mbox/%3
> > > > > CCAB%2
> > > > > BcfdyPDpbUh5FyDT%3D9mU%3DFCHEA1AZpkF6X0nN1t4mjwqu2tA%
> > 40mail.gmail.com%
> > > > > 3E
> > > > >
> > > > > I wonder if there were any helpful suggestions that did not go into
> > > > > the mailing list.
> > > > >
> > > > > I am mostly concerned about restoring data in a Zookeeper ensemble.
> > > > >
> > > > > There is no document at the project web-site that would explain the
> > > > > restore procedure for a distributed deployment. The home-grown
> > > > > solution that involves stopping the whole cluster, wiping out
> > > > > databases on all but one server, restoring the database on one
> > server,
> > > > > and then bring up the cluster and pray that the populated server
> > > > > becomes the leader and populates the cluster. Such solution seems
> to
> > be
> > > > too error-prone.
> > > > >
> > > > > Does anyone have recommendations on how to make it robust?
> > > > >
> > > > > Maybe there is a way to force-populate the ensemble remotely?
> > > > >
> > > > >
> > > > > Thanks,
> > > > > /Sergey
> > > > >
> > > > >
> > > >
> > > >
> > >
> >
>
Re: Backup/restore of an emsemble
Posted by Sergey Maslyakov <ev...@gmail.com>.
Thank you for the reference, Edward! It looks like Guano mostly
accomplished what I want out of it. I spent a little time today digging in
the guts of ZK and found a few ways to make taking a backup more efficient
by delegating all the work to the server. Description of these ways does
not fit the boundaries of a user mailing list and I intend to send a little
write-up into the "dev" mailing list. Please let me know if you are
interested in the subject and I can CC you.
On Fri, Jul 5, 2013 at 12:43 PM, Edward Ribeiro <ed...@gmail.com>wrote:
> Sergey,
>
> Have you looked into the Guano Project (https://github.com/d2fn/guano) ?
> It
> seems to do exactly what you want, or at least as close a dump/restore so
> that you can start from there.
>
> Edward
>
>
> On Fri, Jul 5, 2013 at 1:40 PM, Sergey Maslyakov <ev...@gmail.com>
> wrote:
>
> > The nature of the system that I am working with is mostly read-heavy.
> > Writes happen rarely. In he event of disaster recovery to some point in
> > time in the past, the system should sustain minor inconsistencies.
> >
> > I am also considering to have Zookeeper server create a copy of the
> > DataTree, and then serialize it into a file, which will later be picked
> up
> > by the import client. This could be the most efficient way of taking a
> > backup.
> >
> >
> >
> > On Fri, Jul 5, 2013 at 11:33 AM, Flavio Junqueira <fpjunqueira@yahoo.com
> > >wrote:
> >
> > > In your approach, would you "lock" the zookeeper state and read the
> data
> > > tree using getData/getChildren? If you have concurrent updates, then
> you
> > > may
> > > end up having an inconsistent snapshot.
> > >
> > > -Flavio
> > >
> > > -----Original Message-----
> > > From: Sergey Maslyakov [mailto:evolvah@gmail.com]
> > > Sent: 05 July 2013 17:12
> > > To: user@zookeeper.apache.org
> > > Subject: Re: Backup/restore of an emsemble
> > >
> > > Yes, Flavio, I looked at Exhibitor, but I need a pretty granular
> control
> > > over a cluster of ZK servers. This is why I'm inclined to build
> something
> > > by
> > > hand. So far, a pair of external export and import clients seems like a
> > > promising approach.
> > >
> > > Export would connect to the ensemble and dump out the data into a file
> on
> > > disk. Import would connect, wipe out the namespace, and then reload the
> > > data
> > > from the file that was earlier created by the export client.
> > >
> > >
> > >
> > > On Fri, Jul 5, 2013 at 8:31 AM, Flavio Junqueira
> > > <fp...@yahoo.com>wrote:
> > >
> > > > Sergey,
> > > >
> > > > Have you had a look at Exhibitor?
> > > >
> > > > https://github.com/Netflix/exhibitor
> > > >
> > > > -Flavio
> > > >
> > > > -----Original Message-----
> > > > From: Sergey Maslyakov [mailto:evolvah@gmail.com]
> > > > Sent: 05 July 2013 04:39
> > > > To: user@zookeeper.apache.org
> > > > Subject: Backup/restore of an emsemble
> > > >
> > > > A while ago, Jack Ma asked this question:
> > > >
> > > >
> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201306.mbox/%3
> > > > CCAB%2
> > > > BcfdyPDpbUh5FyDT%3D9mU%3DFCHEA1AZpkF6X0nN1t4mjwqu2tA%
> 40mail.gmail.com%
> > > > 3E
> > > >
> > > > I wonder if there were any helpful suggestions that did not go into
> > > > the mailing list.
> > > >
> > > > I am mostly concerned about restoring data in a Zookeeper ensemble.
> > > >
> > > > There is no document at the project web-site that would explain the
> > > > restore procedure for a distributed deployment. The home-grown
> > > > solution that involves stopping the whole cluster, wiping out
> > > > databases on all but one server, restoring the database on one
> server,
> > > > and then bring up the cluster and pray that the populated server
> > > > becomes the leader and populates the cluster. Such solution seems to
> be
> > > too error-prone.
> > > >
> > > > Does anyone have recommendations on how to make it robust?
> > > >
> > > > Maybe there is a way to force-populate the ensemble remotely?
> > > >
> > > >
> > > > Thanks,
> > > > /Sergey
> > > >
> > > >
> > >
> > >
> >
>
Re: Backup/restore of an emsemble
Posted by Edward Ribeiro <ed...@gmail.com>.
Sergey,
Have you looked into the Guano Project (https://github.com/d2fn/guano) ? It
seems to do exactly what you want, or at least as close a dump/restore so
that you can start from there.
Edward
On Fri, Jul 5, 2013 at 1:40 PM, Sergey Maslyakov <ev...@gmail.com> wrote:
> The nature of the system that I am working with is mostly read-heavy.
> Writes happen rarely. In he event of disaster recovery to some point in
> time in the past, the system should sustain minor inconsistencies.
>
> I am also considering to have Zookeeper server create a copy of the
> DataTree, and then serialize it into a file, which will later be picked up
> by the import client. This could be the most efficient way of taking a
> backup.
>
>
>
> On Fri, Jul 5, 2013 at 11:33 AM, Flavio Junqueira <fpjunqueira@yahoo.com
> >wrote:
>
> > In your approach, would you "lock" the zookeeper state and read the data
> > tree using getData/getChildren? If you have concurrent updates, then you
> > may
> > end up having an inconsistent snapshot.
> >
> > -Flavio
> >
> > -----Original Message-----
> > From: Sergey Maslyakov [mailto:evolvah@gmail.com]
> > Sent: 05 July 2013 17:12
> > To: user@zookeeper.apache.org
> > Subject: Re: Backup/restore of an emsemble
> >
> > Yes, Flavio, I looked at Exhibitor, but I need a pretty granular control
> > over a cluster of ZK servers. This is why I'm inclined to build something
> > by
> > hand. So far, a pair of external export and import clients seems like a
> > promising approach.
> >
> > Export would connect to the ensemble and dump out the data into a file on
> > disk. Import would connect, wipe out the namespace, and then reload the
> > data
> > from the file that was earlier created by the export client.
> >
> >
> >
> > On Fri, Jul 5, 2013 at 8:31 AM, Flavio Junqueira
> > <fp...@yahoo.com>wrote:
> >
> > > Sergey,
> > >
> > > Have you had a look at Exhibitor?
> > >
> > > https://github.com/Netflix/exhibitor
> > >
> > > -Flavio
> > >
> > > -----Original Message-----
> > > From: Sergey Maslyakov [mailto:evolvah@gmail.com]
> > > Sent: 05 July 2013 04:39
> > > To: user@zookeeper.apache.org
> > > Subject: Backup/restore of an emsemble
> > >
> > > A while ago, Jack Ma asked this question:
> > >
> > > http://mail-archives.apache.org/mod_mbox/zookeeper-user/201306.mbox/%3
> > > CCAB%2
> > > BcfdyPDpbUh5FyDT%3D9mU%3DFCHEA1AZpkF6X0nN1t4mjwqu2tA%40mail.gmail.com%
> > > 3E
> > >
> > > I wonder if there were any helpful suggestions that did not go into
> > > the mailing list.
> > >
> > > I am mostly concerned about restoring data in a Zookeeper ensemble.
> > >
> > > There is no document at the project web-site that would explain the
> > > restore procedure for a distributed deployment. The home-grown
> > > solution that involves stopping the whole cluster, wiping out
> > > databases on all but one server, restoring the database on one server,
> > > and then bring up the cluster and pray that the populated server
> > > becomes the leader and populates the cluster. Such solution seems to be
> > too error-prone.
> > >
> > > Does anyone have recommendations on how to make it robust?
> > >
> > > Maybe there is a way to force-populate the ensemble remotely?
> > >
> > >
> > > Thanks,
> > > /Sergey
> > >
> > >
> >
> >
>
Re: Backup/restore of an emsemble
Posted by Sergey Maslyakov <ev...@gmail.com>.
The nature of the system that I am working with is mostly read-heavy.
Writes happen rarely. In he event of disaster recovery to some point in
time in the past, the system should sustain minor inconsistencies.
I am also considering to have Zookeeper server create a copy of the
DataTree, and then serialize it into a file, which will later be picked up
by the import client. This could be the most efficient way of taking a
backup.
On Fri, Jul 5, 2013 at 11:33 AM, Flavio Junqueira <fp...@yahoo.com>wrote:
> In your approach, would you "lock" the zookeeper state and read the data
> tree using getData/getChildren? If you have concurrent updates, then you
> may
> end up having an inconsistent snapshot.
>
> -Flavio
>
> -----Original Message-----
> From: Sergey Maslyakov [mailto:evolvah@gmail.com]
> Sent: 05 July 2013 17:12
> To: user@zookeeper.apache.org
> Subject: Re: Backup/restore of an emsemble
>
> Yes, Flavio, I looked at Exhibitor, but I need a pretty granular control
> over a cluster of ZK servers. This is why I'm inclined to build something
> by
> hand. So far, a pair of external export and import clients seems like a
> promising approach.
>
> Export would connect to the ensemble and dump out the data into a file on
> disk. Import would connect, wipe out the namespace, and then reload the
> data
> from the file that was earlier created by the export client.
>
>
>
> On Fri, Jul 5, 2013 at 8:31 AM, Flavio Junqueira
> <fp...@yahoo.com>wrote:
>
> > Sergey,
> >
> > Have you had a look at Exhibitor?
> >
> > https://github.com/Netflix/exhibitor
> >
> > -Flavio
> >
> > -----Original Message-----
> > From: Sergey Maslyakov [mailto:evolvah@gmail.com]
> > Sent: 05 July 2013 04:39
> > To: user@zookeeper.apache.org
> > Subject: Backup/restore of an emsemble
> >
> > A while ago, Jack Ma asked this question:
> >
> > http://mail-archives.apache.org/mod_mbox/zookeeper-user/201306.mbox/%3
> > CCAB%2
> > BcfdyPDpbUh5FyDT%3D9mU%3DFCHEA1AZpkF6X0nN1t4mjwqu2tA%40mail.gmail.com%
> > 3E
> >
> > I wonder if there were any helpful suggestions that did not go into
> > the mailing list.
> >
> > I am mostly concerned about restoring data in a Zookeeper ensemble.
> >
> > There is no document at the project web-site that would explain the
> > restore procedure for a distributed deployment. The home-grown
> > solution that involves stopping the whole cluster, wiping out
> > databases on all but one server, restoring the database on one server,
> > and then bring up the cluster and pray that the populated server
> > becomes the leader and populates the cluster. Such solution seems to be
> too error-prone.
> >
> > Does anyone have recommendations on how to make it robust?
> >
> > Maybe there is a way to force-populate the ensemble remotely?
> >
> >
> > Thanks,
> > /Sergey
> >
> >
>
>
RE: Backup/restore of an emsemble
Posted by Flavio Junqueira <fp...@yahoo.com>.
In your approach, would you "lock" the zookeeper state and read the data
tree using getData/getChildren? If you have concurrent updates, then you may
end up having an inconsistent snapshot.
-Flavio
-----Original Message-----
From: Sergey Maslyakov [mailto:evolvah@gmail.com]
Sent: 05 July 2013 17:12
To: user@zookeeper.apache.org
Subject: Re: Backup/restore of an emsemble
Yes, Flavio, I looked at Exhibitor, but I need a pretty granular control
over a cluster of ZK servers. This is why I'm inclined to build something by
hand. So far, a pair of external export and import clients seems like a
promising approach.
Export would connect to the ensemble and dump out the data into a file on
disk. Import would connect, wipe out the namespace, and then reload the data
from the file that was earlier created by the export client.
On Fri, Jul 5, 2013 at 8:31 AM, Flavio Junqueira
<fp...@yahoo.com>wrote:
> Sergey,
>
> Have you had a look at Exhibitor?
>
> https://github.com/Netflix/exhibitor
>
> -Flavio
>
> -----Original Message-----
> From: Sergey Maslyakov [mailto:evolvah@gmail.com]
> Sent: 05 July 2013 04:39
> To: user@zookeeper.apache.org
> Subject: Backup/restore of an emsemble
>
> A while ago, Jack Ma asked this question:
>
> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201306.mbox/%3
> CCAB%2
> BcfdyPDpbUh5FyDT%3D9mU%3DFCHEA1AZpkF6X0nN1t4mjwqu2tA%40mail.gmail.com%
> 3E
>
> I wonder if there were any helpful suggestions that did not go into
> the mailing list.
>
> I am mostly concerned about restoring data in a Zookeeper ensemble.
>
> There is no document at the project web-site that would explain the
> restore procedure for a distributed deployment. The home-grown
> solution that involves stopping the whole cluster, wiping out
> databases on all but one server, restoring the database on one server,
> and then bring up the cluster and pray that the populated server
> becomes the leader and populates the cluster. Such solution seems to be
too error-prone.
>
> Does anyone have recommendations on how to make it robust?
>
> Maybe there is a way to force-populate the ensemble remotely?
>
>
> Thanks,
> /Sergey
>
>
Re: Backup/restore of an emsemble
Posted by Sergey Maslyakov <ev...@gmail.com>.
Yes, Flavio, I looked at Exhibitor, but I need a pretty granular control
over a cluster of ZK servers. This is why I'm inclined to build something
by hand. So far, a pair of external export and import clients seems like a
promising approach.
Export would connect to the ensemble and dump out the data into a file on
disk. Import would connect, wipe out the namespace, and then reload the
data from the file that was earlier created by the export client.
On Fri, Jul 5, 2013 at 8:31 AM, Flavio Junqueira <fp...@yahoo.com>wrote:
> Sergey,
>
> Have you had a look at Exhibitor?
>
> https://github.com/Netflix/exhibitor
>
> -Flavio
>
> -----Original Message-----
> From: Sergey Maslyakov [mailto:evolvah@gmail.com]
> Sent: 05 July 2013 04:39
> To: user@zookeeper.apache.org
> Subject: Backup/restore of an emsemble
>
> A while ago, Jack Ma asked this question:
>
> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201306.mbox/%3CCAB%2
> BcfdyPDpbUh5FyDT%3D9mU%3DFCHEA1AZpkF6X0nN1t4mjwqu2tA%40mail.gmail.com%3E
>
> I wonder if there were any helpful suggestions that did not go into the
> mailing list.
>
> I am mostly concerned about restoring data in a Zookeeper ensemble.
>
> There is no document at the project web-site that would explain the restore
> procedure for a distributed deployment. The home-grown solution that
> involves stopping the whole cluster, wiping out databases on all but one
> server, restoring the database on one server, and then bring up the cluster
> and pray that the populated server becomes the leader and populates the
> cluster. Such solution seems to be too error-prone.
>
> Does anyone have recommendations on how to make it robust?
>
> Maybe there is a way to force-populate the ensemble remotely?
>
>
> Thanks,
> /Sergey
>
>
RE: Backup/restore of an emsemble
Posted by Flavio Junqueira <fp...@yahoo.com>.
Sergey,
Have you had a look at Exhibitor?
https://github.com/Netflix/exhibitor
-Flavio
-----Original Message-----
From: Sergey Maslyakov [mailto:evolvah@gmail.com]
Sent: 05 July 2013 04:39
To: user@zookeeper.apache.org
Subject: Backup/restore of an emsemble
A while ago, Jack Ma asked this question:
http://mail-archives.apache.org/mod_mbox/zookeeper-user/201306.mbox/%3CCAB%2
BcfdyPDpbUh5FyDT%3D9mU%3DFCHEA1AZpkF6X0nN1t4mjwqu2tA%40mail.gmail.com%3E
I wonder if there were any helpful suggestions that did not go into the
mailing list.
I am mostly concerned about restoring data in a Zookeeper ensemble.
There is no document at the project web-site that would explain the restore
procedure for a distributed deployment. The home-grown solution that
involves stopping the whole cluster, wiping out databases on all but one
server, restoring the database on one server, and then bring up the cluster
and pray that the populated server becomes the leader and populates the
cluster. Such solution seems to be too error-prone.
Does anyone have recommendations on how to make it robust?
Maybe there is a way to force-populate the ensemble remotely?
Thanks,
/Sergey