You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Qian Ye <ye...@gmail.com> on 2010/03/01 07:18:50 UTC
About load balance in Zookeeper server
Hi guys:
As I know, when a client connected to Zookeeper servers, it would choose a
server randomly (without the zoo_deterministic_conn_odrder on), and then,
the client would talk to the server until a failure happened. It seems that
zookeeper server cannot handle the client connection dynamically according
to the load of the server. If some flaw of a client made the client connect
Zookeeper servers frequently, it may prevent other normal clients from
getting services from Zookeeper, right? So, is there any method to resolve
these two practical problems:
1. Handle and apportion clients dynamically, so every servers would have
balanced load.
2. Some of frequency controller, which set a frequency threshold on the
frequency of requests from a client, prevent server resource from being
exhausted by a few clients.
--
With Regards!
Ye, Qian
Re: About load balance in Zookeeper server
Posted by Qian Ye <ye...@gmail.com>.
thx mahadev :-)
On Thu, Mar 4, 2010 at 4:05 AM, Mahadev Konar <ma...@yahoo-inc.com> wrote:
> Hi Qian,
> I am not sure if I did respond to your email or not. Sorry, too many
> emails
> I am catching up on. You are right that if you specify just a single host
> then the client would not be able to switch to another server. There have
> been some ideas around Dynamic configuration and storing zookeeper ensemble
> information on the zookeeper cluster itself.
>
> http://issues.apache.org/jira/browse/ZOOKEEPER-338
> http://issues.apache.org/jira/browse/ZOOKEEPER-107
> http://issues.apache.org/jira/browse/ZOOKEEPER-390
>
> This might answer some of the problems you mention, but they are all being
> worked upon!
>
> Thanks
> mahadev
>
>
> On 3/1/10 6:09 PM, "Qian Ye" <ye...@gmail.com> wrote:
>
> > Thanks Mahadev, I see what you mean.
> >
> > Here is another question, the client need a list of Zookeeper servers to
> > initialize the handler, and there is no API for the client to get
> awareness
> > of all the Zookeeper servers in one cluster. That means, if I only
> provide
> > one Zookeeper server in the client's host list, the client would not
> switch
> > to another available Zookeeper server, when the given one was failed. I
> > think is strategy is flawed. The client should be able to find out all
> the
> > Zookeeper servers in the cluster. Is there any compromise for this issue?
> >
> > thanks
> >
> > On Tue, Mar 2, 2010 at 7:29 AM, Mahadev Konar <ma...@yahoo-inc.com>
> wrote:
> >
> >> HI Qian,
> >> You are right we do have any way of handling clients dynamically so that
> >> every server has balanced load. This requires a careful design since we
> >> would not want client connections to keep flipping around and also
> maintain
> >> stability as much as we can. We have had some discussions about it but
> >> nothing concrete has materialized yet.
> >>
> >> We do have checks in place that prevent more than a certain number of
> >> connections (default 10) from the same ip address. This is to keep too
> many
> >> zookeeper client instances from the same client bogging down the
> zookeeper
> >> service. Also, we have throttling for number of outstanding requests
> from
> >> clients (currently set to 1000 by default). This allows zookeeper
> service
> >> to
> >> throttle zookeeper clients. This throttling isnt done on per client
> basis
> >> but is just a check to not bring down the zookeeper service because of
> some
> >> misbehaved client.
> >> Any other checks that you specifically were thinking of?
> >>
> >> Thanks
> >> mahadev
> >>
> >> On 2/28/10 10:18 PM, "Qian Ye" <ye...@gmail.com> wrote:
> >>
> >>> Hi guys:
> >>>
> >>> As I know, when a client connected to Zookeeper servers, it would
> choose
> >> a
> >>> server randomly (without the zoo_deterministic_conn_odrder on), and
> then,
> >>> the client would talk to the server until a failure happened. It seems
> >> that
> >>> zookeeper server cannot handle the client connection dynamically
> >> according
> >>> to the load of the server. If some flaw of a client made the client
> >> connect
> >>> Zookeeper servers frequently, it may prevent other normal clients from
> >>> getting services from Zookeeper, right? So, is there any method to
> >> resolve
> >>> these two practical problems:
> >>>
> >>> 1. Handle and apportion clients dynamically, so every servers would
> have
> >>> balanced load.
> >>> 2. Some of frequency controller, which set a frequency threshold on the
> >>> frequency of requests from a client, prevent server resource from being
> >>> exhausted by a few clients.
> >>>
> >>> --
> >>> With Regards!
> >>>
> >>> Ye, Qian
> >>
> >>
> >
>
>
--
With Regards!
Ye, Qian
Re: About load balance in Zookeeper server
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Qian,
I am not sure if I did respond to your email or not. Sorry, too many emails
I am catching up on. You are right that if you specify just a single host
then the client would not be able to switch to another server. There have
been some ideas around Dynamic configuration and storing zookeeper ensemble
information on the zookeeper cluster itself.
http://issues.apache.org/jira/browse/ZOOKEEPER-338
http://issues.apache.org/jira/browse/ZOOKEEPER-107
http://issues.apache.org/jira/browse/ZOOKEEPER-390
This might answer some of the problems you mention, but they are all being
worked upon!
Thanks
mahadev
On 3/1/10 6:09 PM, "Qian Ye" <ye...@gmail.com> wrote:
> Thanks Mahadev, I see what you mean.
>
> Here is another question, the client need a list of Zookeeper servers to
> initialize the handler, and there is no API for the client to get awareness
> of all the Zookeeper servers in one cluster. That means, if I only provide
> one Zookeeper server in the client's host list, the client would not switch
> to another available Zookeeper server, when the given one was failed. I
> think is strategy is flawed. The client should be able to find out all the
> Zookeeper servers in the cluster. Is there any compromise for this issue?
>
> thanks
>
> On Tue, Mar 2, 2010 at 7:29 AM, Mahadev Konar <ma...@yahoo-inc.com> wrote:
>
>> HI Qian,
>> You are right we do have any way of handling clients dynamically so that
>> every server has balanced load. This requires a careful design since we
>> would not want client connections to keep flipping around and also maintain
>> stability as much as we can. We have had some discussions about it but
>> nothing concrete has materialized yet.
>>
>> We do have checks in place that prevent more than a certain number of
>> connections (default 10) from the same ip address. This is to keep too many
>> zookeeper client instances from the same client bogging down the zookeeper
>> service. Also, we have throttling for number of outstanding requests from
>> clients (currently set to 1000 by default). This allows zookeeper service
>> to
>> throttle zookeeper clients. This throttling isnt done on per client basis
>> but is just a check to not bring down the zookeeper service because of some
>> misbehaved client.
>> Any other checks that you specifically were thinking of?
>>
>> Thanks
>> mahadev
>>
>> On 2/28/10 10:18 PM, "Qian Ye" <ye...@gmail.com> wrote:
>>
>>> Hi guys:
>>>
>>> As I know, when a client connected to Zookeeper servers, it would choose
>> a
>>> server randomly (without the zoo_deterministic_conn_odrder on), and then,
>>> the client would talk to the server until a failure happened. It seems
>> that
>>> zookeeper server cannot handle the client connection dynamically
>> according
>>> to the load of the server. If some flaw of a client made the client
>> connect
>>> Zookeeper servers frequently, it may prevent other normal clients from
>>> getting services from Zookeeper, right? So, is there any method to
>> resolve
>>> these two practical problems:
>>>
>>> 1. Handle and apportion clients dynamically, so every servers would have
>>> balanced load.
>>> 2. Some of frequency controller, which set a frequency threshold on the
>>> frequency of requests from a client, prevent server resource from being
>>> exhausted by a few clients.
>>>
>>> --
>>> With Regards!
>>>
>>> Ye, Qian
>>
>>
>
Re: About load balance in Zookeeper server
Posted by Qian Ye <ye...@gmail.com>.
Thanks Mahadev, I see what you mean.
Here is another question, the client need a list of Zookeeper servers to
initialize the handler, and there is no API for the client to get awareness
of all the Zookeeper servers in one cluster. That means, if I only provide
one Zookeeper server in the client's host list, the client would not switch
to another available Zookeeper server, when the given one was failed. I
think is strategy is flawed. The client should be able to find out all the
Zookeeper servers in the cluster. Is there any compromise for this issue?
thanks
On Tue, Mar 2, 2010 at 7:29 AM, Mahadev Konar <ma...@yahoo-inc.com> wrote:
> HI Qian,
> You are right we do have any way of handling clients dynamically so that
> every server has balanced load. This requires a careful design since we
> would not want client connections to keep flipping around and also maintain
> stability as much as we can. We have had some discussions about it but
> nothing concrete has materialized yet.
>
> We do have checks in place that prevent more than a certain number of
> connections (default 10) from the same ip address. This is to keep too many
> zookeeper client instances from the same client bogging down the zookeeper
> service. Also, we have throttling for number of outstanding requests from
> clients (currently set to 1000 by default). This allows zookeeper service
> to
> throttle zookeeper clients. This throttling isnt done on per client basis
> but is just a check to not bring down the zookeeper service because of some
> misbehaved client.
> Any other checks that you specifically were thinking of?
>
> Thanks
> mahadev
>
> On 2/28/10 10:18 PM, "Qian Ye" <ye...@gmail.com> wrote:
>
> > Hi guys:
> >
> > As I know, when a client connected to Zookeeper servers, it would choose
> a
> > server randomly (without the zoo_deterministic_conn_odrder on), and then,
> > the client would talk to the server until a failure happened. It seems
> that
> > zookeeper server cannot handle the client connection dynamically
> according
> > to the load of the server. If some flaw of a client made the client
> connect
> > Zookeeper servers frequently, it may prevent other normal clients from
> > getting services from Zookeeper, right? So, is there any method to
> resolve
> > these two practical problems:
> >
> > 1. Handle and apportion clients dynamically, so every servers would have
> > balanced load.
> > 2. Some of frequency controller, which set a frequency threshold on the
> > frequency of requests from a client, prevent server resource from being
> > exhausted by a few clients.
> >
> > --
> > With Regards!
> >
> > Ye, Qian
>
>
--
With Regards!
Ye, Qian
Re: About load balance in Zookeeper server
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
HI Qian,
You are right we do have any way of handling clients dynamically so that
every server has balanced load. This requires a careful design since we
would not want client connections to keep flipping around and also maintain
stability as much as we can. We have had some discussions about it but
nothing concrete has materialized yet.
We do have checks in place that prevent more than a certain number of
connections (default 10) from the same ip address. This is to keep too many
zookeeper client instances from the same client bogging down the zookeeper
service. Also, we have throttling for number of outstanding requests from
clients (currently set to 1000 by default). This allows zookeeper service to
throttle zookeeper clients. This throttling isnt done on per client basis
but is just a check to not bring down the zookeeper service because of some
misbehaved client.
Any other checks that you specifically were thinking of?
Thanks
mahadev
On 2/28/10 10:18 PM, "Qian Ye" <ye...@gmail.com> wrote:
> Hi guys:
>
> As I know, when a client connected to Zookeeper servers, it would choose a
> server randomly (without the zoo_deterministic_conn_odrder on), and then,
> the client would talk to the server until a failure happened. It seems that
> zookeeper server cannot handle the client connection dynamically according
> to the load of the server. If some flaw of a client made the client connect
> Zookeeper servers frequently, it may prevent other normal clients from
> getting services from Zookeeper, right? So, is there any method to resolve
> these two practical problems:
>
> 1. Handle and apportion clients dynamically, so every servers would have
> balanced load.
> 2. Some of frequency controller, which set a frequency threshold on the
> frequency of requests from a client, prevent server resource from being
> exhausted by a few clients.
>
> --
> With Regards!
>
> Ye, Qian