You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by "Kiesslich, Thomas" <th...@siemens.com> on 2008/11/17 11:46:02 UTC
Dynamic server management?
Hi all,
I am evaluating ZooKeeper as a discovery protocol. I want to change the number of servers that build up ensemble during runtime of an existing number servers. Is that possible or do I have to change the configuration and restart all servers? I found that if you have 3 servers configured and startup the first it is continously looking for the at least one other server. Is it really necessary to do that? If the first server does not find anyone else, it could elect itself as the leader. If the next server starts up the can redo the election I guess.
Could you please help me with these questions? Thank you.
Mit freundlichen Grüßen / With best regards
Thomas Kießlich
Siemens Enterprise Communications GmbH & Co. KG
HiPath Applications
SEN LIP DA 11
Schertlinstr. 8
81379 Munich, Germany
Phone: +49 (89) 722-32483
Fax: +49 (89) 722-40560
Email: thomas.kiesslich@siemens.com
Communication for the open minded
www.siemens.de/open
www.siemens.com/open
Siemens Enterprise Communications GmbH & Co. KG
Managing Directors: Andreas Bernhardt, Reinhard Benditte,
Gerhard Otterbach, Thomas Zimmermann
Registered offices: Munich
Commercial registry Munich, HRA 88546
WEEE-Reg.-No. DE 27980375
General Partner:
Siemens Enterprise Communications Management GmbH
Registered offices: Munich
Commercial registry: Munich, HRB 163415
Wichtiger Hinweis: Diese Information ist für den Gebrauch durch die Person oder die Firma/Organisation bestimmt, die in der Empfängeradresse benannt ist. Wenn Sie nicht der angegebene Empfänger sind, nehmen Sie bitte zur Kenntnis, daß Weitergabe, Kopieren, Verteilung oder Nutzung des Inhalts dieser Email-Übertragung unzulässig ist. Falls Sie diese Email irrtümlich erhalten haben, benachrichtigen Sie den Absender bitte unverzüglich telefonisch oder durch eine Email. Vielen Dank.
Important Note: This e-mail may contain trade secrets or privileged, undisclosed or otherwise confidential information. If you have received this e-mail in error, you are hereby notified that any review, copying or distribution of it is strictly prohibited. Please inform us immediately and destroy the original transmittal. Thank you for your cooperation.
Re: ZooKeeper Roadmap - 3.1.0 and beyond.
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Krishna,
Sorry for the delayed response. The responses are in line.
On 11/18/08 12:02 PM, "Krishna Sankar (ksankar)" <ks...@cisco.com> wrote:
> Have a couple of questions on the proposed multi-tenancy feature (pardon
> me if they are obvious, as I am slowly getting up to speed):
>
> a) First, good initiative. I think this will make ZK more
> pervasive. I plan to participate and contribute
thanks and we look forward to your contribution.
> b) Is there any assumption on the trust and security ? i.e. could
> we assume that the servers would be in a secure environment and so no
> need for SSL et al., could we trust the MAC Address/IP Address (this
> also raises the question of NAT et al, if they are relevant) and could
> we make an assumption that there is no need for a secure identity ?
There is an assumption to some level. We do trust the ipaddress (got via
tcp connections assuming its difficult to forge) and we use
Raw tcp (so no security in the transfer layer). We do have a authorization
layer that is pluggable at the server and the clients can identify
themselves using that.
> c) I remember seeing one of Ben's ToDo, an entry for distributed
> ZK. I couldn't find a resolution or write-up. Possible, I am missing
> something obvious. Anyway, Is it already in place or do we need to
> consider that feature in the multi-tenancy capability ?
Its still in discussions. We don't have a conrete proposal yet. For multi
tenancy we don't need to consider it.
mahadev
>
> Cheers & thanks
> <k/>
>
> |-----Original Message-----
> |From: Benjamin Patrick Hunt <ph...@apache.org>
> |Sent: Mon, 27 Oct 2008 20:35:37 -0700
> |To: zookeeper-user@hadoop.apache.org
> |Subject: ZooKeeper Roadmap - 3.1.0 and beyond.
> |
> <..snip/>
> 5) (begin) multi-tenancy support. A number of users have expressed
> interest in being able to deploy ZK as a service in a cloud.
> Multi-tenancy support would be a huge benefit (quota, qos, namespace
> partitioning of nodes, billing, etc...)
>
> <..snip>
>
Re: ZooKeeper Roadmap - 3.1.0 and beyond.
Posted by "Krishna Sankar (ksankar)" <ks...@cisco.com>.
Have a couple of questions on the proposed multi-tenancy feature (pardon
me if they are obvious, as I am slowly getting up to speed):
a) First, good initiative. I think this will make ZK more
pervasive. I plan to participate and contribute
b) Is there any assumption on the trust and security ? i.e. could
we assume that the servers would be in a secure environment and so no
need for SSL et al., could we trust the MAC Address/IP Address (this
also raises the question of NAT et al, if they are relevant) and could
we make an assumption that there is no need for a secure identity ?
c) I remember seeing one of Ben's ToDo, an entry for distributed
ZK. I couldn't find a resolution or write-up. Possible, I am missing
something obvious. Anyway, Is it already in place or do we need to
consider that feature in the multi-tenancy capability ?
Cheers & thanks
<k/>
|-----Original Message-----
|From: Benjamin Patrick Hunt <ph...@apache.org>
|Sent: Mon, 27 Oct 2008 20:35:37 -0700
|To: zookeeper-user@hadoop.apache.org
|Subject: ZooKeeper Roadmap - 3.1.0 and beyond.
|
<..snip/>
5) (begin) multi-tenancy support. A number of users have expressed
interest in being able to deploy ZK as a service in a cloud.
Multi-tenancy support would be a huge benefit (quota, qos, namespace
partitioning of nodes, billing, etc...)
<..snip>
RE: Watches are one time trigger?
Posted by Benjamin Reed <br...@yahoo-inc.com>.
Watches also do not have any data associated with them, so even if they were not one-time, you would not get any extra information and you have the potential of generating a lot of extra network traffic.
ben
-----Original Message-----
From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
Sent: Tuesday, November 18, 2008 8:38 AM
To: zookeeper-user@hadoop.apache.org
Subject: Re: Watches are one time trigger?
Hi Thomas,
Yes, watches are one time triggers. Watches are supposed to be lighweight
and local to the server you are connected to.
We designed it such that it has minimum load impact on the servers.
mahadev
On 11/18/08 5:58 AM, "Kiesslich, Thomas" <th...@siemens.com>
wrote:
>
> Hi,
>
> A watch is designed as a one time trigger. Why have you designed it that way?
> Why not as a normal listener?
>
>
> Mit freundlichen Grüßen / With best regards
> Thomas Kießlich
>
> Siemens Enterprise Communications GmbH & Co. KG
> HiPath Applications
>
> SEN LIP DA 11
> Schertlinstr. 8
> 81379 Munich, Germany
Re: Watches are one time trigger?
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Thomas,
Yes, watches are one time triggers. Watches are supposed to be lighweight
and local to the server you are connected to.
We designed it such that it has minimum load impact on the servers.
mahadev
On 11/18/08 5:58 AM, "Kiesslich, Thomas" <th...@siemens.com>
wrote:
>
> Hi,
>
> A watch is designed as a one time trigger. Why have you designed it that way?
> Why not as a normal listener?
>
>
> Mit freundlichen Grüßen / With best regards
> Thomas Kießlich
>
> Siemens Enterprise Communications GmbH & Co. KG
> HiPath Applications
>
> SEN LIP DA 11
> Schertlinstr. 8
> 81379 Munich, Germany
Watches are one time trigger?
Posted by "Kiesslich, Thomas" <th...@siemens.com>.
Hi,
A watch is designed as a one time trigger. Why have you designed it that way? Why not as a normal listener?
Mit freundlichen Grüßen / With best regards
Thomas Kießlich
Siemens Enterprise Communications GmbH & Co. KG
HiPath Applications
SEN LIP DA 11
Schertlinstr. 8
81379 Munich, Germany
RE: Dynamic server management?
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
Hi Thomas, In the scenario you describe, 1 and 2 won't make any progress as
they are not part of a component that contains a quorum (a majority in this
current implementation). Servers 3,4,5 will elect a leader (if they don't
have one yet), and will make progress independently. Once the segmentation
heals, 1 and 2 will synchronize with the current leader.
Thanks,
-Flavio
> -----Original Message-----
> From: Kiesslich, Thomas [mailto:thomas.kiesslich@siemens.com]
> Sent: Monday, November 17, 2008 2:04 PM
> To: zookeeper-user@hadoop.apache.org
> Subject: AW: Dynamic server management?
>
> Hi Flavio,
>
> Thank you for the information.
>
> So how do you handle than the following scenario?
> You have 5 servers and temporary network segmentation, where server 1 and
> 2 cannot talk to 3,4 and 5. Will server 1 and 2 do a new leader election
> for their partition, as well server 3,4 and 5 for their partition? If yes,
> how do they synchronize if they can reach each other again?
>
>
> Mit freundlichen Grüßen / With best regards
> Thomas Kießlich
>
> Siemens Enterprise Communications GmbH & Co. KG
> HiPath Applications
>
> SEN LIP DA 11
> Schertlinstr. 8
> 81379 Munich, Germany
>
> -----Ursprüngliche Nachricht-----
> Von: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> Gesendet: Montag, 17. November 2008 13:49
> An: zookeeper-user@hadoop.apache.org
> Betreff: RE: Dynamic server management?
>
> Hi Thomas, We currently don't have such a feature of adding and removing
> servers dynamically, although we would like to, so we'll have it
> eventually.
>
> Without a dynamic mechanism for adding and removing servers, your example
> is
> problematic. Suppose that you configure your ensemble to have 3 servers,
> and
> then you start only one. Your single server would say: "Hey, I'm alone so
> I
> should make progress independently". This is reasonable, right? Well, not
> really, because you can have the following alternative execution. You
> start
> your three servers, but there is a temporary network partition and server
> 1
> cannot talk to servers 2 and 3. Consequently, server 1 will make progress
> independently of 2 and 3, and their state will diverge.
>
> Hope it helps.
>
> -Flavio
>
> > -----Original Message-----
> > From: Kiesslich, Thomas [mailto:thomas.kiesslich@siemens.com]
> > Sent: Monday, November 17, 2008 11:46 AM
> > To: zookeeper-user@hadoop.apache.org
> > Subject: Dynamic server management?
> >
> > Hi all,
> >
> > I am evaluating ZooKeeper as a discovery protocol. I want to change the
> > number of servers that build up ensemble during runtime of an existing
> > number servers. Is that possible or do I have to change the
> configuration
> > and restart all servers? I found that if you have 3 servers configured
> and
> > startup the first it is continously looking for the at least one other
> > server. Is it really necessary to do that? If the first server does not
> > find anyone else, it could elect itself as the leader. If the next
> server
> > starts up the can redo the election I guess.
> > Could you please help me with these questions? Thank you.
> >
> >
> >
> > Mit freundlichen Grüßen / With best regards
> > Thomas Kießlich
> >
> > Siemens Enterprise Communications GmbH & Co. KG
> > HiPath Applications
> >
> > SEN LIP DA 11
> > Schertlinstr. 8
> > 81379 Munich, Germany
> >
> > Phone: +49 (89) 722-32483
> > Fax: +49 (89) 722-40560
> > Email: thomas.kiesslich@siemens.com
> >
> > Communication for the open minded
> > www.siemens.de/open
> > www.siemens.com/open
> >
> > Siemens Enterprise Communications GmbH & Co. KG
> > Managing Directors: Andreas Bernhardt, Reinhard Benditte,
> > Gerhard Otterbach, Thomas Zimmermann
> > Registered offices: Munich
> > Commercial registry Munich, HRA 88546
> > WEEE-Reg.-No. DE 27980375
> > General Partner:
> > Siemens Enterprise Communications Management GmbH
> > Registered offices: Munich
> > Commercial registry: Munich, HRB 163415
> >
> > Wichtiger Hinweis: Diese Information ist für den Gebrauch durch die
> Person
> > oder die Firma/Organisation bestimmt, die in der Empfängeradresse
> benannt
> > ist. Wenn Sie nicht der angegebene Empfänger sind, nehmen Sie bitte zur
> > Kenntnis, daß Weitergabe, Kopieren, Verteilung oder Nutzung des Inhalts
> > dieser Email-Übertragung unzulässig ist. Falls Sie diese Email
> irrtümlich
> > erhalten haben, benachrichtigen Sie den Absender bitte unverzüglich
> > telefonisch oder durch eine Email. Vielen Dank.
> >
> > Important Note: This e-mail may contain trade secrets or privileged,
> > undisclosed or otherwise confidential information. If you have received
> > this e-mail in error, you are hereby notified that any review, copying
> or
> > distribution of it is strictly prohibited. Please inform us immediately
> > and destroy the original transmittal. Thank you for your cooperation.
> >
>
AW: Dynamic server management?
Posted by "Kiesslich, Thomas" <th...@siemens.com>.
Hi Flavio,
Thank you for the information.
So how do you handle than the following scenario?
You have 5 servers and temporary network segmentation, where server 1 and 2 cannot talk to 3,4 and 5. Will server 1 and 2 do a new leader election for their partition, as well server 3,4 and 5 for their partition? If yes, how do they synchronize if they can reach each other again?
Mit freundlichen Grüßen / With best regards
Thomas Kießlich
Siemens Enterprise Communications GmbH & Co. KG
HiPath Applications
SEN LIP DA 11
Schertlinstr. 8
81379 Munich, Germany
-----Ursprüngliche Nachricht-----
Von: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
Gesendet: Montag, 17. November 2008 13:49
An: zookeeper-user@hadoop.apache.org
Betreff: RE: Dynamic server management?
Hi Thomas, We currently don't have such a feature of adding and removing
servers dynamically, although we would like to, so we'll have it eventually.
Without a dynamic mechanism for adding and removing servers, your example is
problematic. Suppose that you configure your ensemble to have 3 servers, and
then you start only one. Your single server would say: "Hey, I'm alone so I
should make progress independently". This is reasonable, right? Well, not
really, because you can have the following alternative execution. You start
your three servers, but there is a temporary network partition and server 1
cannot talk to servers 2 and 3. Consequently, server 1 will make progress
independently of 2 and 3, and their state will diverge.
Hope it helps.
-Flavio
> -----Original Message-----
> From: Kiesslich, Thomas [mailto:thomas.kiesslich@siemens.com]
> Sent: Monday, November 17, 2008 11:46 AM
> To: zookeeper-user@hadoop.apache.org
> Subject: Dynamic server management?
>
> Hi all,
>
> I am evaluating ZooKeeper as a discovery protocol. I want to change the
> number of servers that build up ensemble during runtime of an existing
> number servers. Is that possible or do I have to change the configuration
> and restart all servers? I found that if you have 3 servers configured and
> startup the first it is continously looking for the at least one other
> server. Is it really necessary to do that? If the first server does not
> find anyone else, it could elect itself as the leader. If the next server
> starts up the can redo the election I guess.
> Could you please help me with these questions? Thank you.
>
>
>
> Mit freundlichen Grüßen / With best regards
> Thomas Kießlich
>
> Siemens Enterprise Communications GmbH & Co. KG
> HiPath Applications
>
> SEN LIP DA 11
> Schertlinstr. 8
> 81379 Munich, Germany
>
> Phone: +49 (89) 722-32483
> Fax: +49 (89) 722-40560
> Email: thomas.kiesslich@siemens.com
>
> Communication for the open minded
> www.siemens.de/open
> www.siemens.com/open
>
> Siemens Enterprise Communications GmbH & Co. KG
> Managing Directors: Andreas Bernhardt, Reinhard Benditte,
> Gerhard Otterbach, Thomas Zimmermann
> Registered offices: Munich
> Commercial registry Munich, HRA 88546
> WEEE-Reg.-No. DE 27980375
> General Partner:
> Siemens Enterprise Communications Management GmbH
> Registered offices: Munich
> Commercial registry: Munich, HRB 163415
>
> Wichtiger Hinweis: Diese Information ist für den Gebrauch durch die Person
> oder die Firma/Organisation bestimmt, die in der Empfängeradresse benannt
> ist. Wenn Sie nicht der angegebene Empfänger sind, nehmen Sie bitte zur
> Kenntnis, daß Weitergabe, Kopieren, Verteilung oder Nutzung des Inhalts
> dieser Email-Übertragung unzulässig ist. Falls Sie diese Email irrtümlich
> erhalten haben, benachrichtigen Sie den Absender bitte unverzüglich
> telefonisch oder durch eine Email. Vielen Dank.
>
> Important Note: This e-mail may contain trade secrets or privileged,
> undisclosed or otherwise confidential information. If you have received
> this e-mail in error, you are hereby notified that any review, copying or
> distribution of it is strictly prohibited. Please inform us immediately
> and destroy the original transmittal. Thank you for your cooperation.
>
RE: Dynamic server management?
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
Hi Thomas, We currently don't have such a feature of adding and removing
servers dynamically, although we would like to, so we'll have it eventually.
Without a dynamic mechanism for adding and removing servers, your example is
problematic. Suppose that you configure your ensemble to have 3 servers, and
then you start only one. Your single server would say: "Hey, I'm alone so I
should make progress independently". This is reasonable, right? Well, not
really, because you can have the following alternative execution. You start
your three servers, but there is a temporary network partition and server 1
cannot talk to servers 2 and 3. Consequently, server 1 will make progress
independently of 2 and 3, and their state will diverge.
Hope it helps.
-Flavio
> -----Original Message-----
> From: Kiesslich, Thomas [mailto:thomas.kiesslich@siemens.com]
> Sent: Monday, November 17, 2008 11:46 AM
> To: zookeeper-user@hadoop.apache.org
> Subject: Dynamic server management?
>
> Hi all,
>
> I am evaluating ZooKeeper as a discovery protocol. I want to change the
> number of servers that build up ensemble during runtime of an existing
> number servers. Is that possible or do I have to change the configuration
> and restart all servers? I found that if you have 3 servers configured and
> startup the first it is continously looking for the at least one other
> server. Is it really necessary to do that? If the first server does not
> find anyone else, it could elect itself as the leader. If the next server
> starts up the can redo the election I guess.
> Could you please help me with these questions? Thank you.
>
>
>
> Mit freundlichen Grüßen / With best regards
> Thomas Kießlich
>
> Siemens Enterprise Communications GmbH & Co. KG
> HiPath Applications
>
> SEN LIP DA 11
> Schertlinstr. 8
> 81379 Munich, Germany
>
> Phone: +49 (89) 722-32483
> Fax: +49 (89) 722-40560
> Email: thomas.kiesslich@siemens.com
>
> Communication for the open minded
> www.siemens.de/open
> www.siemens.com/open
>
> Siemens Enterprise Communications GmbH & Co. KG
> Managing Directors: Andreas Bernhardt, Reinhard Benditte,
> Gerhard Otterbach, Thomas Zimmermann
> Registered offices: Munich
> Commercial registry Munich, HRA 88546
> WEEE-Reg.-No. DE 27980375
> General Partner:
> Siemens Enterprise Communications Management GmbH
> Registered offices: Munich
> Commercial registry: Munich, HRB 163415
>
> Wichtiger Hinweis: Diese Information ist für den Gebrauch durch die Person
> oder die Firma/Organisation bestimmt, die in der Empfängeradresse benannt
> ist. Wenn Sie nicht der angegebene Empfänger sind, nehmen Sie bitte zur
> Kenntnis, daß Weitergabe, Kopieren, Verteilung oder Nutzung des Inhalts
> dieser Email-Übertragung unzulässig ist. Falls Sie diese Email irrtümlich
> erhalten haben, benachrichtigen Sie den Absender bitte unverzüglich
> telefonisch oder durch eine Email. Vielen Dank.
>
> Important Note: This e-mail may contain trade secrets or privileged,
> undisclosed or otherwise confidential information. If you have received
> this e-mail in error, you are hereby notified that any review, copying or
> distribution of it is strictly prohibited. Please inform us immediately
> and destroy the original transmittal. Thank you for your cooperation.
>