You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Tobias Schulze-Heinrichs <t....@xplosion.de> on 2012/01/05 21:34:59 UTC
Kafka managed by zookeeper
Hello all,
I have an ensemble of 3 ZK Server. Do I need to configure all 3 zookeeper servers to my Kafka server or is there a possibility that it can get the address of 2 ZK out of one which is given at my Kafka configuration?
Best regards
Tobias
Re: Kafka managed by zookeeper
Posted by Neha Narkhede <ne...@gmail.com>.
>> is this an issue which is not implemented in Kafka or does zookeeper doesn't offer this information?
As far as I know, this is how the zookeeper client works today. It has
to know the list of hosts that constitute a zookeeper cluster (though
the connection url), it caches the IPs locally, and uses plain round
robin to pick the next server to connect to.
Thanks,
Neha
On Fri, Jan 6, 2012 at 2:16 AM, Pierre-Yves Ritschard <py...@spootnik.org> wrote:
> Well, I am running zookeeper on EC2 and rely on configuration
> management to update configuration files when topology changes. Chef
> and Pallet in particular are very good at addressing that type of
> problem.
> As for haproxy shifting the problem, you can rely on elastic ips and
> two load-balancing instances. If you want to keep the moving parts and
> types of nodes down and don't already run a load-balancing service,
> the CM approach is your best bet
>
> On Fri, Jan 6, 2012 at 11:04 AM, Tobias Schulze-Heinrichs
> <t....@xplosion.de> wrote:
>> Hmm, that shift the problem just to another resource. If the haproxy instance crashes I'll need to reconfigure the producer and consumers. I guess that problem needs to be addressed to the experts of cloud computing.
>>
>> Beside that I read that the zookeeper guys just working on a way to dynamically add and/or remove zookeeper servers to an existing ensemble.
>>
>> Best regards
>> Tobias
>>
>>
>> Am 06.01.2012 um 09:18 schrieb Pierre-Yves Ritschard:
>>
>>> or you could load-balance access to your ensemble using haproxy or elb
>>>
>>> On Fri, Jan 6, 2012 at 9:06 AM, Tobias Schulze-Heinrichs
>>> <t....@xplosion.de> wrote:
>>>> Hi Neha,
>>>>
>>>> is this an issue which is not implemented in Kafka or does zookeeper doesn't offer this information? To describe this problem a little bit more detailed - we want to set up a service at the cloud (amazon ec2). Therefor it might occur that one instance crashes and came up with a new IP. In that scenario I have to reconfigure and restart all Zookeeper and all Kafka server, right?
>>>>
>>>> Best regards
>>>> Tobias
>>>>
>>>>
>>>> Am 05.01.2012 um 21:39 schrieb Neha Narkhede:
>>>>
>>>>> Tobias,
>>>>>
>>>>> Please configure Kafka with the list of all 3 zookeeper servers. The
>>>>> way the zookeeper client works is it just picks one zookeeper server
>>>>> from the list of all zookeeper servers in the connection url. If you
>>>>> miss specifying one server in your connection url, Kafka will never
>>>>> connect to the 3rd server, causing imbalance on the zookeeper cluster.
>>>>>
>>>>> Thanks,
>>>>> Neha
>>>>>
>>>>> On Thu, Jan 5, 2012 at 12:34 PM, Tobias Schulze-Heinrichs
>>>>> <t....@xplosion.de> wrote:
>>>>>> Hello all,
>>>>>>
>>>>>> I have an ensemble of 3 ZK Server. Do I need to configure all 3 zookeeper servers to my Kafka server or is there a possibility that it can get the address of 2 ZK out of one which is given at my Kafka configuration?
>>>>>>
>>>>>> Best regards
>>>>>> Tobias
>>>>
>>
Re: Kafka managed by zookeeper
Posted by Pierre-Yves Ritschard <py...@spootnik.org>.
Well, I am running zookeeper on EC2 and rely on configuration
management to update configuration files when topology changes. Chef
and Pallet in particular are very good at addressing that type of
problem.
As for haproxy shifting the problem, you can rely on elastic ips and
two load-balancing instances. If you want to keep the moving parts and
types of nodes down and don't already run a load-balancing service,
the CM approach is your best bet
On Fri, Jan 6, 2012 at 11:04 AM, Tobias Schulze-Heinrichs
<t....@xplosion.de> wrote:
> Hmm, that shift the problem just to another resource. If the haproxy instance crashes I'll need to reconfigure the producer and consumers. I guess that problem needs to be addressed to the experts of cloud computing.
>
> Beside that I read that the zookeeper guys just working on a way to dynamically add and/or remove zookeeper servers to an existing ensemble.
>
> Best regards
> Tobias
>
>
> Am 06.01.2012 um 09:18 schrieb Pierre-Yves Ritschard:
>
>> or you could load-balance access to your ensemble using haproxy or elb
>>
>> On Fri, Jan 6, 2012 at 9:06 AM, Tobias Schulze-Heinrichs
>> <t....@xplosion.de> wrote:
>>> Hi Neha,
>>>
>>> is this an issue which is not implemented in Kafka or does zookeeper doesn't offer this information? To describe this problem a little bit more detailed - we want to set up a service at the cloud (amazon ec2). Therefor it might occur that one instance crashes and came up with a new IP. In that scenario I have to reconfigure and restart all Zookeeper and all Kafka server, right?
>>>
>>> Best regards
>>> Tobias
>>>
>>>
>>> Am 05.01.2012 um 21:39 schrieb Neha Narkhede:
>>>
>>>> Tobias,
>>>>
>>>> Please configure Kafka with the list of all 3 zookeeper servers. The
>>>> way the zookeeper client works is it just picks one zookeeper server
>>>> from the list of all zookeeper servers in the connection url. If you
>>>> miss specifying one server in your connection url, Kafka will never
>>>> connect to the 3rd server, causing imbalance on the zookeeper cluster.
>>>>
>>>> Thanks,
>>>> Neha
>>>>
>>>> On Thu, Jan 5, 2012 at 12:34 PM, Tobias Schulze-Heinrichs
>>>> <t....@xplosion.de> wrote:
>>>>> Hello all,
>>>>>
>>>>> I have an ensemble of 3 ZK Server. Do I need to configure all 3 zookeeper servers to my Kafka server or is there a possibility that it can get the address of 2 ZK out of one which is given at my Kafka configuration?
>>>>>
>>>>> Best regards
>>>>> Tobias
>>>
>
Re: Kafka managed by zookeeper
Posted by Tobias Schulze-Heinrichs <t....@xplosion.de>.
Hmm, that shift the problem just to another resource. If the haproxy instance crashes I'll need to reconfigure the producer and consumers. I guess that problem needs to be addressed to the experts of cloud computing.
Beside that I read that the zookeeper guys just working on a way to dynamically add and/or remove zookeeper servers to an existing ensemble.
Best regards
Tobias
Am 06.01.2012 um 09:18 schrieb Pierre-Yves Ritschard:
> or you could load-balance access to your ensemble using haproxy or elb
>
> On Fri, Jan 6, 2012 at 9:06 AM, Tobias Schulze-Heinrichs
> <t....@xplosion.de> wrote:
>> Hi Neha,
>>
>> is this an issue which is not implemented in Kafka or does zookeeper doesn't offer this information? To describe this problem a little bit more detailed - we want to set up a service at the cloud (amazon ec2). Therefor it might occur that one instance crashes and came up with a new IP. In that scenario I have to reconfigure and restart all Zookeeper and all Kafka server, right?
>>
>> Best regards
>> Tobias
>>
>>
>> Am 05.01.2012 um 21:39 schrieb Neha Narkhede:
>>
>>> Tobias,
>>>
>>> Please configure Kafka with the list of all 3 zookeeper servers. The
>>> way the zookeeper client works is it just picks one zookeeper server
>>> from the list of all zookeeper servers in the connection url. If you
>>> miss specifying one server in your connection url, Kafka will never
>>> connect to the 3rd server, causing imbalance on the zookeeper cluster.
>>>
>>> Thanks,
>>> Neha
>>>
>>> On Thu, Jan 5, 2012 at 12:34 PM, Tobias Schulze-Heinrichs
>>> <t....@xplosion.de> wrote:
>>>> Hello all,
>>>>
>>>> I have an ensemble of 3 ZK Server. Do I need to configure all 3 zookeeper servers to my Kafka server or is there a possibility that it can get the address of 2 ZK out of one which is given at my Kafka configuration?
>>>>
>>>> Best regards
>>>> Tobias
>>
Re: Kafka managed by zookeeper
Posted by Pierre-Yves Ritschard <py...@spootnik.org>.
or you could load-balance access to your ensemble using haproxy or elb
On Fri, Jan 6, 2012 at 9:06 AM, Tobias Schulze-Heinrichs
<t....@xplosion.de> wrote:
> Hi Neha,
>
> is this an issue which is not implemented in Kafka or does zookeeper doesn't offer this information? To describe this problem a little bit more detailed - we want to set up a service at the cloud (amazon ec2). Therefor it might occur that one instance crashes and came up with a new IP. In that scenario I have to reconfigure and restart all Zookeeper and all Kafka server, right?
>
> Best regards
> Tobias
>
>
> Am 05.01.2012 um 21:39 schrieb Neha Narkhede:
>
>> Tobias,
>>
>> Please configure Kafka with the list of all 3 zookeeper servers. The
>> way the zookeeper client works is it just picks one zookeeper server
>> from the list of all zookeeper servers in the connection url. If you
>> miss specifying one server in your connection url, Kafka will never
>> connect to the 3rd server, causing imbalance on the zookeeper cluster.
>>
>> Thanks,
>> Neha
>>
>> On Thu, Jan 5, 2012 at 12:34 PM, Tobias Schulze-Heinrichs
>> <t....@xplosion.de> wrote:
>>> Hello all,
>>>
>>> I have an ensemble of 3 ZK Server. Do I need to configure all 3 zookeeper servers to my Kafka server or is there a possibility that it can get the address of 2 ZK out of one which is given at my Kafka configuration?
>>>
>>> Best regards
>>> Tobias
>
Re: Kafka managed by zookeeper
Posted by Tobias Schulze-Heinrichs <t....@xplosion.de>.
Hi Neha,
is this an issue which is not implemented in Kafka or does zookeeper doesn't offer this information? To describe this problem a little bit more detailed - we want to set up a service at the cloud (amazon ec2). Therefor it might occur that one instance crashes and came up with a new IP. In that scenario I have to reconfigure and restart all Zookeeper and all Kafka server, right?
Best regards
Tobias
Am 05.01.2012 um 21:39 schrieb Neha Narkhede:
> Tobias,
>
> Please configure Kafka with the list of all 3 zookeeper servers. The
> way the zookeeper client works is it just picks one zookeeper server
> from the list of all zookeeper servers in the connection url. If you
> miss specifying one server in your connection url, Kafka will never
> connect to the 3rd server, causing imbalance on the zookeeper cluster.
>
> Thanks,
> Neha
>
> On Thu, Jan 5, 2012 at 12:34 PM, Tobias Schulze-Heinrichs
> <t....@xplosion.de> wrote:
>> Hello all,
>>
>> I have an ensemble of 3 ZK Server. Do I need to configure all 3 zookeeper servers to my Kafka server or is there a possibility that it can get the address of 2 ZK out of one which is given at my Kafka configuration?
>>
>> Best regards
>> Tobias
Re: Kafka managed by zookeeper
Posted by Neha Narkhede <ne...@gmail.com>.
Tobias,
Please configure Kafka with the list of all 3 zookeeper servers. The
way the zookeeper client works is it just picks one zookeeper server
from the list of all zookeeper servers in the connection url. If you
miss specifying one server in your connection url, Kafka will never
connect to the 3rd server, causing imbalance on the zookeeper cluster.
Thanks,
Neha
On Thu, Jan 5, 2012 at 12:34 PM, Tobias Schulze-Heinrichs
<t....@xplosion.de> wrote:
> Hello all,
>
> I have an ensemble of 3 ZK Server. Do I need to configure all 3 zookeeper servers to my Kafka server or is there a possibility that it can get the address of 2 ZK out of one which is given at my Kafka configuration?
>
> Best regards
> Tobias