You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@helix.apache.org by Erwin Karbasi <er...@optinity.com> on 2013/06/13 22:43:53 UTC
Is zookeeper for Helix has standalone JAVA process?
Hello Maters,
We don't want to have separate/specific JAVA process for Zookeeper.
As I figured out from the Master-Slave sample we can start the Zookeeper
programmatically using the Zookeeper API.
I have several questions:
1. If we are using the Zookeeper API for initializing the Zookeeper from
the Helix app (out app that uses Helix) would we have specific process for
Zookeeper? Would we have one process for our app and another for the
Zookeeper?
What happens if we'll not initialize the Zookeeper from the Helix app
programmatically but using the commands, whether then would we have
specific process for Zookeeper?
2. If we'll decide to initialize Zookeeper from Helix app,
programmatically, How can we use Zookeeper clustering (ensemble) , is there
any API for Zookeeper clustering, any direction would highly appreciated.
3. Is it possible to start the Zookeeper by using commands and not
initializing specific process for it but using the Helix app (our app that
uses Helix API)?
I hope my questions are enough clear.
Thanks in advance,
Erwin Karbasi
AT&T, Senior Software Architect
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by Erwin Karbasi <er...@optinity.com>.
Hi Kishore,
Do you have any documentation/sample regarding zookeeper implementation
programmatically?
Thanks,
Erwin
On Jun 15, 2013 11:28 AM, "Erwin Karbasi" <er...@optinity.com> wrote:
> Hi kishore,
>
> Thank you for preservance and helpful support.
>
> Do you have any direction how can we implement zookeeper clustering
> programmatically in my aforementioned suggestion?
> As far as I know we need to use quorum. Do you know any example that can
> help us.
>
> Thanks,
> Erwin
> On Jun 15, 2013 3:46 AM, "kishore g" <g....@gmail.com> wrote:
>
>> Hi Erwin,
>>
>> Yes thats possible. Good idea.
>>
>> thanks,
>> Kishore G
>>
>>
>> On Fri, Jun 14, 2013 at 3:42 PM, Erwin Karbasi <er...@optinity.com>wrote:
>>
>>> Kishore,
>>>
>>> Thank you a lot for detailed explanation.
>>>
>>> I thought to define in configuration of each participant whether it will
>>> act as controller or Zookeeper as well. All participants will know those
>>> controllers and zookeepers that defined.
>>> For example I will define in 5 participants that they also should act as
>>> controller and zookeepers as well on the same process of the participants.
>>> Is it possible?
>>>
>>> Cheers,
>>> Erwin
>>> On Jun 14, 2013 6:21 PM, "kishore g" <g....@gmail.com> wrote:
>>>
>>>> Hi Erwin,
>>>>
>>>> Here are the options:
>>>>
>>>> 1. participant controller and zookeeper on one process on all 50 boxes:
>>>> This is not possible, you need only 3 or 5 zookeepers running and all
>>>> participant/controllers need to know the machinename: port of the zookeeper
>>>> boxes upfront.
>>>>
>>>> 2. separate zookeeper but participant and controller in one process:
>>>> This is possible, controller can be started as part of all participants but
>>>> only one controller will be active. This is fine until you have like 100
>>>> nodes, after which its better to have a separate controller.
>>>>
>>>> 3. zookeeper and controller on same box but separate process,
>>>> participants in different: This is possible, note that once you this
>>>> configuration all additional clusters you deploy can share the same
>>>> zookeeper/controller. So if you plan to have many clusters this is a better
>>>> option and much easier to manage the clusters. You can use standard
>>>> commands to start zookeeper and helix controller.
>>>>
>>>> 4. zookeeper and controller on same box and in one process: This also
>>>> works and the advantages of 3 holds good here as well. You will have to
>>>> start
>>>> both of them programmatically. Any time you restart a process, both
>>>> zookeeper and controller will be upgraded.
>>>>
>>>>
>>>> My preference is 3) 4) 2), you cannot do 1).
>>>>
>>>> I agree with you that it requires additional monitoring but actual
>>>> experience suggests that co-locating zookeeper with participants causes
>>>> un-wanted problems. For example, the participants are heavily loaded and
>>>> your application might have large GC pauses/consume al cpu/memory on that
>>>> box. This will adversely impact the functionality of zookeeper.
>>>>
>>>> Thanks,
>>>> Kishore G
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Jun 14, 2013 at 5:44 AM, Erwin Karbasi <er...@optinity.com>wrote:
>>>>
>>>>> My question is, whether is it possible to collocate participant,
>>>>> controller and zookeeper on one process, is there any limitations in term
>>>>> of Helix or Zookeeper?
>>>>> Any direction regarding above approach would be appreciated.
>>>>> On Jun 14, 2013 3:23 PM, "Ming Fang" <mi...@mac.com> wrote:
>>>>>
>>>>>> That's not possible at this time.
>>>>>> ZAC currently must run as a standalone process and only allow one
>>>>>> instance.
>>>>>>
>>>>>> Sent from my iPad
>>>>>>
>>>>>> On Jun 14, 2013, at 7:35 AM, Erwin Karbasi <er...@optinity.com>
>>>>>> wrote:
>>>>>>
>>>>>> If I have 40 app server instances (JVMs) could I run participants on
>>>>>> all of them, controller and zookeeper just on 5 of then, but on the same
>>>>>> process of the 5 participants?
>>>>>> On Jun 14, 2013 2:29 PM, "Ming Fang" <mi...@mac.com> wrote:
>>>>>>
>>>>>>> ZAC currently runs Zookeeper in standalone mode.
>>>>>>> Therefore you can only run one instance.
>>>>>>> If you also run the participant in it then that limits you to one
>>>>>>> instance of the participant also.
>>>>>>>
>>>>>>>
>>>>>>> Sent from my iPad
>>>>>>>
>>>>>>> On Jun 14, 2013, at 7:23 AM, Erwin Karbasi <er...@optinity.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Ming,
>>>>>>>
>>>>>>> Thank you for fast turnaround response.
>>>>>>>
>>>>>>> Could I run my app that uses Helix (participant), controller and
>>>>>>> zookeeper on the same process?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Erwin
>>>>>>> On Jun 14, 2013 2:16 PM, "Ming Fang" <mi...@mac.com> wrote:
>>>>>>>
>>>>>>>> Erwin
>>>>>>>>
>>>>>>>> Here is an example of running Zookeeper, and Controller in the same
>>>>>>>> process,
>>>>>>>>
>>>>>>>> https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java
>>>>>>>> We have this in production.
>>>>>>>>
>>>>>>>> Sent from my iPad
>>>>>>>>
>>>>>>>> On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <er...@optinity.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I know that it's better that participants, controller and zookeeper
>>>>>>>> will have separate java processes but we have more then 40 boxes with app
>>>>>>>> server that we'd like to manage them with Helix Leader election and cannot
>>>>>>>> add more processes because each additional java process requires management
>>>>>>>> (monitoring, alarms, KPI, etc...), so we prefer to avoid the additional
>>>>>>>> java processes. We'll not manage to approve additional processes
>>>>>>>>
>>>>>>>> Your insight would appreciated.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Erwin
>>>>>>>> On Jun 14, 2013 9:58 AM, "kishore g" <g....@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Erwin,
>>>>>>>>>
>>>>>>>>> This is how I would setup your cluster. Allocate 3 boxes that only
>>>>>>>>> run zookeeper and Helix controller. Its better that they are two separate
>>>>>>>>> process. These boxes need not be powerful but isolating zookeeper and helix
>>>>>>>>> controller from actual server is a good deployment practice. See
>>>>>>>>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetupon how to use zookeeper clustering ensemble.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Run the real servers on the rest of the boxes. Do not start
>>>>>>>>> zookeeper as part of these processes, instead simply provide the zookeeper
>>>>>>>>> address (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Kishore G
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <
>>>>>>>>> erwin@optinity.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Kishore,
>>>>>>>>>>
>>>>>>>>>> We have more then 50 nodes.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Erwin
>>>>>>>>>> On Jun 14, 2013 1:51 AM, "kishore g" <g....@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Erwin,
>>>>>>>>>>>
>>>>>>>>>>> How many nodes do you have in your cluster ?
>>>>>>>>>>>
>>>>>>>>>>> thanks,
>>>>>>>>>>> Kishore G
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <
>>>>>>>>>>> erwin@optinity.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello Maters,
>>>>>>>>>>>>
>>>>>>>>>>>> We don't want to have separate/specific JAVA process for
>>>>>>>>>>>> Zookeeper.
>>>>>>>>>>>> As I figured out from the Master-Slave sample we can start the
>>>>>>>>>>>> Zookeeper programmatically using the Zookeeper API.
>>>>>>>>>>>>
>>>>>>>>>>>> I have several questions:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. If we are using the Zookeeper API for initializing the
>>>>>>>>>>>> Zookeeper from the Helix app (out app that uses Helix) would we have
>>>>>>>>>>>> specific process for Zookeeper? Would we have one process for our app and
>>>>>>>>>>>> another for the Zookeeper?
>>>>>>>>>>>> What happens if we'll not initialize the Zookeeper from the
>>>>>>>>>>>> Helix app programmatically but using the commands, whether then would we
>>>>>>>>>>>> have specific process for Zookeeper?
>>>>>>>>>>>>
>>>>>>>>>>>> 2. If we'll decide to initialize Zookeeper from Helix app,
>>>>>>>>>>>> programmatically, How can we use Zookeeper clustering (ensemble) , is there
>>>>>>>>>>>> any API for Zookeeper clustering, any direction would highly appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>> 3. Is it possible to start the Zookeeper by using commands and
>>>>>>>>>>>> not initializing specific process for it but using the Helix app (our app
>>>>>>>>>>>> that uses Helix API)?
>>>>>>>>>>>>
>>>>>>>>>>>> I hope my questions are enough clear.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>> Erwin Karbasi
>>>>>>>>>>>> AT&T, Senior Software Architect
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>
>>
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by Erwin Karbasi <er...@optinity.com>.
Hi kishore,
Thank you for preservance and helpful support.
Do you have any direction how can we implement zookeeper clustering
programmatically in my aforementioned suggestion?
As far as I know we need to use quorum. Do you know any example that can
help us.
Thanks,
Erwin
On Jun 15, 2013 3:46 AM, "kishore g" <g....@gmail.com> wrote:
> Hi Erwin,
>
> Yes thats possible. Good idea.
>
> thanks,
> Kishore G
>
>
> On Fri, Jun 14, 2013 at 3:42 PM, Erwin Karbasi <er...@optinity.com> wrote:
>
>> Kishore,
>>
>> Thank you a lot for detailed explanation.
>>
>> I thought to define in configuration of each participant whether it will
>> act as controller or Zookeeper as well. All participants will know those
>> controllers and zookeepers that defined.
>> For example I will define in 5 participants that they also should act as
>> controller and zookeepers as well on the same process of the participants.
>> Is it possible?
>>
>> Cheers,
>> Erwin
>> On Jun 14, 2013 6:21 PM, "kishore g" <g....@gmail.com> wrote:
>>
>>> Hi Erwin,
>>>
>>> Here are the options:
>>>
>>> 1. participant controller and zookeeper on one process on all 50 boxes:
>>> This is not possible, you need only 3 or 5 zookeepers running and all
>>> participant/controllers need to know the machinename: port of the zookeeper
>>> boxes upfront.
>>>
>>> 2. separate zookeeper but participant and controller in one process:
>>> This is possible, controller can be started as part of all participants but
>>> only one controller will be active. This is fine until you have like 100
>>> nodes, after which its better to have a separate controller.
>>>
>>> 3. zookeeper and controller on same box but separate process,
>>> participants in different: This is possible, note that once you this
>>> configuration all additional clusters you deploy can share the same
>>> zookeeper/controller. So if you plan to have many clusters this is a better
>>> option and much easier to manage the clusters. You can use standard
>>> commands to start zookeeper and helix controller.
>>>
>>> 4. zookeeper and controller on same box and in one process: This also
>>> works and the advantages of 3 holds good here as well. You will have to
>>> start
>>> both of them programmatically. Any time you restart a process, both
>>> zookeeper and controller will be upgraded.
>>>
>>>
>>> My preference is 3) 4) 2), you cannot do 1).
>>>
>>> I agree with you that it requires additional monitoring but actual
>>> experience suggests that co-locating zookeeper with participants causes
>>> un-wanted problems. For example, the participants are heavily loaded and
>>> your application might have large GC pauses/consume al cpu/memory on that
>>> box. This will adversely impact the functionality of zookeeper.
>>>
>>> Thanks,
>>> Kishore G
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Jun 14, 2013 at 5:44 AM, Erwin Karbasi <er...@optinity.com>wrote:
>>>
>>>> My question is, whether is it possible to collocate participant,
>>>> controller and zookeeper on one process, is there any limitations in term
>>>> of Helix or Zookeeper?
>>>> Any direction regarding above approach would be appreciated.
>>>> On Jun 14, 2013 3:23 PM, "Ming Fang" <mi...@mac.com> wrote:
>>>>
>>>>> That's not possible at this time.
>>>>> ZAC currently must run as a standalone process and only allow one
>>>>> instance.
>>>>>
>>>>> Sent from my iPad
>>>>>
>>>>> On Jun 14, 2013, at 7:35 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>>>>
>>>>> If I have 40 app server instances (JVMs) could I run participants on
>>>>> all of them, controller and zookeeper just on 5 of then, but on the same
>>>>> process of the 5 participants?
>>>>> On Jun 14, 2013 2:29 PM, "Ming Fang" <mi...@mac.com> wrote:
>>>>>
>>>>>> ZAC currently runs Zookeeper in standalone mode.
>>>>>> Therefore you can only run one instance.
>>>>>> If you also run the participant in it then that limits you to one
>>>>>> instance of the participant also.
>>>>>>
>>>>>>
>>>>>> Sent from my iPad
>>>>>>
>>>>>> On Jun 14, 2013, at 7:23 AM, Erwin Karbasi <er...@optinity.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Ming,
>>>>>>
>>>>>> Thank you for fast turnaround response.
>>>>>>
>>>>>> Could I run my app that uses Helix (participant), controller and
>>>>>> zookeeper on the same process?
>>>>>>
>>>>>> Cheers,
>>>>>> Erwin
>>>>>> On Jun 14, 2013 2:16 PM, "Ming Fang" <mi...@mac.com> wrote:
>>>>>>
>>>>>>> Erwin
>>>>>>>
>>>>>>> Here is an example of running Zookeeper, and Controller in the same
>>>>>>> process,
>>>>>>>
>>>>>>> https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java
>>>>>>> We have this in production.
>>>>>>>
>>>>>>> Sent from my iPad
>>>>>>>
>>>>>>> On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <er...@optinity.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> I know that it's better that participants, controller and zookeeper
>>>>>>> will have separate java processes but we have more then 40 boxes with app
>>>>>>> server that we'd like to manage them with Helix Leader election and cannot
>>>>>>> add more processes because each additional java process requires management
>>>>>>> (monitoring, alarms, KPI, etc...), so we prefer to avoid the additional
>>>>>>> java processes. We'll not manage to approve additional processes
>>>>>>>
>>>>>>> Your insight would appreciated.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Erwin
>>>>>>> On Jun 14, 2013 9:58 AM, "kishore g" <g....@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Erwin,
>>>>>>>>
>>>>>>>> This is how I would setup your cluster. Allocate 3 boxes that only
>>>>>>>> run zookeeper and Helix controller. Its better that they are two separate
>>>>>>>> process. These boxes need not be powerful but isolating zookeeper and helix
>>>>>>>> controller from actual server is a good deployment practice. See
>>>>>>>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetupon how to use zookeeper clustering ensemble.
>>>>>>>>
>>>>>>>>
>>>>>>>> Run the real servers on the rest of the boxes. Do not start
>>>>>>>> zookeeper as part of these processes, instead simply provide the zookeeper
>>>>>>>> address (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Kishore G
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <erwin@optinity.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi Kishore,
>>>>>>>>>
>>>>>>>>> We have more then 50 nodes.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Erwin
>>>>>>>>> On Jun 14, 2013 1:51 AM, "kishore g" <g....@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Erwin,
>>>>>>>>>>
>>>>>>>>>> How many nodes do you have in your cluster ?
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>> Kishore G
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <
>>>>>>>>>> erwin@optinity.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello Maters,
>>>>>>>>>>>
>>>>>>>>>>> We don't want to have separate/specific JAVA process for
>>>>>>>>>>> Zookeeper.
>>>>>>>>>>> As I figured out from the Master-Slave sample we can start the
>>>>>>>>>>> Zookeeper programmatically using the Zookeeper API.
>>>>>>>>>>>
>>>>>>>>>>> I have several questions:
>>>>>>>>>>>
>>>>>>>>>>> 1. If we are using the Zookeeper API for initializing the
>>>>>>>>>>> Zookeeper from the Helix app (out app that uses Helix) would we have
>>>>>>>>>>> specific process for Zookeeper? Would we have one process for our app and
>>>>>>>>>>> another for the Zookeeper?
>>>>>>>>>>> What happens if we'll not initialize the Zookeeper from the
>>>>>>>>>>> Helix app programmatically but using the commands, whether then would we
>>>>>>>>>>> have specific process for Zookeeper?
>>>>>>>>>>>
>>>>>>>>>>> 2. If we'll decide to initialize Zookeeper from Helix app,
>>>>>>>>>>> programmatically, How can we use Zookeeper clustering (ensemble) , is there
>>>>>>>>>>> any API for Zookeeper clustering, any direction would highly appreciated.
>>>>>>>>>>>
>>>>>>>>>>> 3. Is it possible to start the Zookeeper by using commands and
>>>>>>>>>>> not initializing specific process for it but using the Helix app (our app
>>>>>>>>>>> that uses Helix API)?
>>>>>>>>>>>
>>>>>>>>>>> I hope my questions are enough clear.
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>> Erwin Karbasi
>>>>>>>>>>> AT&T, Senior Software Architect
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>
>
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by kishore g <g....@gmail.com>.
Hi Erwin,
Yes thats possible. Good idea.
thanks,
Kishore G
On Fri, Jun 14, 2013 at 3:42 PM, Erwin Karbasi <er...@optinity.com> wrote:
> Kishore,
>
> Thank you a lot for detailed explanation.
>
> I thought to define in configuration of each participant whether it will
> act as controller or Zookeeper as well. All participants will know those
> controllers and zookeepers that defined.
> For example I will define in 5 participants that they also should act as
> controller and zookeepers as well on the same process of the participants.
> Is it possible?
>
> Cheers,
> Erwin
> On Jun 14, 2013 6:21 PM, "kishore g" <g....@gmail.com> wrote:
>
>> Hi Erwin,
>>
>> Here are the options:
>>
>> 1. participant controller and zookeeper on one process on all 50 boxes:
>> This is not possible, you need only 3 or 5 zookeepers running and all
>> participant/controllers need to know the machinename: port of the zookeeper
>> boxes upfront.
>>
>> 2. separate zookeeper but participant and controller in one process: This
>> is possible, controller can be started as part of all participants but only
>> one controller will be active. This is fine until you have like 100 nodes,
>> after which its better to have a separate controller.
>>
>> 3. zookeeper and controller on same box but separate process,
>> participants in different: This is possible, note that once you this
>> configuration all additional clusters you deploy can share the same
>> zookeeper/controller. So if you plan to have many clusters this is a better
>> option and much easier to manage the clusters. You can use standard
>> commands to start zookeeper and helix controller.
>>
>> 4. zookeeper and controller on same box and in one process: This also
>> works and the advantages of 3 holds good here as well. You will have to
>> start
>> both of them programmatically. Any time you restart a process, both
>> zookeeper and controller will be upgraded.
>>
>>
>> My preference is 3) 4) 2), you cannot do 1).
>>
>> I agree with you that it requires additional monitoring but actual
>> experience suggests that co-locating zookeeper with participants causes
>> un-wanted problems. For example, the participants are heavily loaded and
>> your application might have large GC pauses/consume al cpu/memory on that
>> box. This will adversely impact the functionality of zookeeper.
>>
>> Thanks,
>> Kishore G
>>
>>
>>
>>
>>
>> On Fri, Jun 14, 2013 at 5:44 AM, Erwin Karbasi <er...@optinity.com>wrote:
>>
>>> My question is, whether is it possible to collocate participant,
>>> controller and zookeeper on one process, is there any limitations in term
>>> of Helix or Zookeeper?
>>> Any direction regarding above approach would be appreciated.
>>> On Jun 14, 2013 3:23 PM, "Ming Fang" <mi...@mac.com> wrote:
>>>
>>>> That's not possible at this time.
>>>> ZAC currently must run as a standalone process and only allow one
>>>> instance.
>>>>
>>>> Sent from my iPad
>>>>
>>>> On Jun 14, 2013, at 7:35 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>>>
>>>> If I have 40 app server instances (JVMs) could I run participants on
>>>> all of them, controller and zookeeper just on 5 of then, but on the same
>>>> process of the 5 participants?
>>>> On Jun 14, 2013 2:29 PM, "Ming Fang" <mi...@mac.com> wrote:
>>>>
>>>>> ZAC currently runs Zookeeper in standalone mode.
>>>>> Therefore you can only run one instance.
>>>>> If you also run the participant in it then that limits you to one
>>>>> instance of the participant also.
>>>>>
>>>>>
>>>>> Sent from my iPad
>>>>>
>>>>> On Jun 14, 2013, at 7:23 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>>>>
>>>>> Hi Ming,
>>>>>
>>>>> Thank you for fast turnaround response.
>>>>>
>>>>> Could I run my app that uses Helix (participant), controller and
>>>>> zookeeper on the same process?
>>>>>
>>>>> Cheers,
>>>>> Erwin
>>>>> On Jun 14, 2013 2:16 PM, "Ming Fang" <mi...@mac.com> wrote:
>>>>>
>>>>>> Erwin
>>>>>>
>>>>>> Here is an example of running Zookeeper, and Controller in the same
>>>>>> process,
>>>>>>
>>>>>> https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java
>>>>>> We have this in production.
>>>>>>
>>>>>> Sent from my iPad
>>>>>>
>>>>>> On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <er...@optinity.com>
>>>>>> wrote:
>>>>>>
>>>>>> I know that it's better that participants, controller and zookeeper
>>>>>> will have separate java processes but we have more then 40 boxes with app
>>>>>> server that we'd like to manage them with Helix Leader election and cannot
>>>>>> add more processes because each additional java process requires management
>>>>>> (monitoring, alarms, KPI, etc...), so we prefer to avoid the additional
>>>>>> java processes. We'll not manage to approve additional processes
>>>>>>
>>>>>> Your insight would appreciated.
>>>>>>
>>>>>> Thanks,
>>>>>> Erwin
>>>>>> On Jun 14, 2013 9:58 AM, "kishore g" <g....@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Erwin,
>>>>>>>
>>>>>>> This is how I would setup your cluster. Allocate 3 boxes that only
>>>>>>> run zookeeper and Helix controller. Its better that they are two separate
>>>>>>> process. These boxes need not be powerful but isolating zookeeper and helix
>>>>>>> controller from actual server is a good deployment practice. See
>>>>>>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetupon how to use zookeeper clustering ensemble.
>>>>>>>
>>>>>>>
>>>>>>> Run the real servers on the rest of the boxes. Do not start
>>>>>>> zookeeper as part of these processes, instead simply provide the zookeeper
>>>>>>> address (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kishore G
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <er...@optinity.com>wrote:
>>>>>>>
>>>>>>>> Hi Kishore,
>>>>>>>>
>>>>>>>> We have more then 50 nodes.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Erwin
>>>>>>>> On Jun 14, 2013 1:51 AM, "kishore g" <g....@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Erwin,
>>>>>>>>>
>>>>>>>>> How many nodes do you have in your cluster ?
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> Kishore G
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <erwin@optinity.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> Hello Maters,
>>>>>>>>>>
>>>>>>>>>> We don't want to have separate/specific JAVA process for
>>>>>>>>>> Zookeeper.
>>>>>>>>>> As I figured out from the Master-Slave sample we can start the
>>>>>>>>>> Zookeeper programmatically using the Zookeeper API.
>>>>>>>>>>
>>>>>>>>>> I have several questions:
>>>>>>>>>>
>>>>>>>>>> 1. If we are using the Zookeeper API for initializing the
>>>>>>>>>> Zookeeper from the Helix app (out app that uses Helix) would we have
>>>>>>>>>> specific process for Zookeeper? Would we have one process for our app and
>>>>>>>>>> another for the Zookeeper?
>>>>>>>>>> What happens if we'll not initialize the Zookeeper from the Helix
>>>>>>>>>> app programmatically but using the commands, whether then would we have
>>>>>>>>>> specific process for Zookeeper?
>>>>>>>>>>
>>>>>>>>>> 2. If we'll decide to initialize Zookeeper from Helix app,
>>>>>>>>>> programmatically, How can we use Zookeeper clustering (ensemble) , is there
>>>>>>>>>> any API for Zookeeper clustering, any direction would highly appreciated.
>>>>>>>>>>
>>>>>>>>>> 3. Is it possible to start the Zookeeper by using commands and
>>>>>>>>>> not initializing specific process for it but using the Helix app (our app
>>>>>>>>>> that uses Helix API)?
>>>>>>>>>>
>>>>>>>>>> I hope my questions are enough clear.
>>>>>>>>>>
>>>>>>>>>> Thanks in advance,
>>>>>>>>>> Erwin Karbasi
>>>>>>>>>> AT&T, Senior Software Architect
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by Erwin Karbasi <er...@optinity.com>.
Kishore,
Thank you a lot for detailed explanation.
I thought to define in configuration of each participant whether it will
act as controller or Zookeeper as well. All participants will know those
controllers and zookeepers that defined.
For example I will define in 5 participants that they also should act as
controller and zookeepers as well on the same process of the participants.
Is it possible?
Cheers,
Erwin
On Jun 14, 2013 6:21 PM, "kishore g" <g....@gmail.com> wrote:
> Hi Erwin,
>
> Here are the options:
>
> 1. participant controller and zookeeper on one process on all 50 boxes:
> This is not possible, you need only 3 or 5 zookeepers running and all
> participant/controllers need to know the machinename: port of the zookeeper
> boxes upfront.
>
> 2. separate zookeeper but participant and controller in one process: This
> is possible, controller can be started as part of all participants but only
> one controller will be active. This is fine until you have like 100 nodes,
> after which its better to have a separate controller.
>
> 3. zookeeper and controller on same box but separate process, participants
> in different: This is possible, note that once you this configuration all
> additional clusters you deploy can share the same zookeeper/controller. So
> if you plan to have many clusters this is a better option and much easier
> to manage the clusters. You can use standard commands to start zookeeper
> and helix controller.
>
> 4. zookeeper and controller on same box and in one process: This also
> works and the advantages of 3 holds good here as well. You will have to
> start
> both of them programmatically. Any time you restart a process, both
> zookeeper and controller will be upgraded.
>
>
> My preference is 3) 4) 2), you cannot do 1).
>
> I agree with you that it requires additional monitoring but actual
> experience suggests that co-locating zookeeper with participants causes
> un-wanted problems. For example, the participants are heavily loaded and
> your application might have large GC pauses/consume al cpu/memory on that
> box. This will adversely impact the functionality of zookeeper.
>
> Thanks,
> Kishore G
>
>
>
>
>
> On Fri, Jun 14, 2013 at 5:44 AM, Erwin Karbasi <er...@optinity.com> wrote:
>
>> My question is, whether is it possible to collocate participant,
>> controller and zookeeper on one process, is there any limitations in term
>> of Helix or Zookeeper?
>> Any direction regarding above approach would be appreciated.
>> On Jun 14, 2013 3:23 PM, "Ming Fang" <mi...@mac.com> wrote:
>>
>>> That's not possible at this time.
>>> ZAC currently must run as a standalone process and only allow one
>>> instance.
>>>
>>> Sent from my iPad
>>>
>>> On Jun 14, 2013, at 7:35 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>>
>>> If I have 40 app server instances (JVMs) could I run participants on all
>>> of them, controller and zookeeper just on 5 of then, but on the same
>>> process of the 5 participants?
>>> On Jun 14, 2013 2:29 PM, "Ming Fang" <mi...@mac.com> wrote:
>>>
>>>> ZAC currently runs Zookeeper in standalone mode.
>>>> Therefore you can only run one instance.
>>>> If you also run the participant in it then that limits you to one
>>>> instance of the participant also.
>>>>
>>>>
>>>> Sent from my iPad
>>>>
>>>> On Jun 14, 2013, at 7:23 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>>>
>>>> Hi Ming,
>>>>
>>>> Thank you for fast turnaround response.
>>>>
>>>> Could I run my app that uses Helix (participant), controller and
>>>> zookeeper on the same process?
>>>>
>>>> Cheers,
>>>> Erwin
>>>> On Jun 14, 2013 2:16 PM, "Ming Fang" <mi...@mac.com> wrote:
>>>>
>>>>> Erwin
>>>>>
>>>>> Here is an example of running Zookeeper, and Controller in the same
>>>>> process,
>>>>>
>>>>> https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java
>>>>> We have this in production.
>>>>>
>>>>> Sent from my iPad
>>>>>
>>>>> On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>>>>
>>>>> I know that it's better that participants, controller and zookeeper
>>>>> will have separate java processes but we have more then 40 boxes with app
>>>>> server that we'd like to manage them with Helix Leader election and cannot
>>>>> add more processes because each additional java process requires management
>>>>> (monitoring, alarms, KPI, etc...), so we prefer to avoid the additional
>>>>> java processes. We'll not manage to approve additional processes
>>>>>
>>>>> Your insight would appreciated.
>>>>>
>>>>> Thanks,
>>>>> Erwin
>>>>> On Jun 14, 2013 9:58 AM, "kishore g" <g....@gmail.com> wrote:
>>>>>
>>>>>> Hi Erwin,
>>>>>>
>>>>>> This is how I would setup your cluster. Allocate 3 boxes that only
>>>>>> run zookeeper and Helix controller. Its better that they are two separate
>>>>>> process. These boxes need not be powerful but isolating zookeeper and helix
>>>>>> controller from actual server is a good deployment practice. See
>>>>>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetupon how to use zookeeper clustering ensemble.
>>>>>>
>>>>>>
>>>>>> Run the real servers on the rest of the boxes. Do not start zookeeper
>>>>>> as part of these processes, instead simply provide the zookeeper address
>>>>>> (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore G
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <er...@optinity.com>wrote:
>>>>>>
>>>>>>> Hi Kishore,
>>>>>>>
>>>>>>> We have more then 50 nodes.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Erwin
>>>>>>> On Jun 14, 2013 1:51 AM, "kishore g" <g....@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Erwin,
>>>>>>>>
>>>>>>>> How many nodes do you have in your cluster ?
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Kishore G
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <er...@optinity.com>wrote:
>>>>>>>>
>>>>>>>>> Hello Maters,
>>>>>>>>>
>>>>>>>>> We don't want to have separate/specific JAVA process for Zookeeper.
>>>>>>>>> As I figured out from the Master-Slave sample we can start the
>>>>>>>>> Zookeeper programmatically using the Zookeeper API.
>>>>>>>>>
>>>>>>>>> I have several questions:
>>>>>>>>>
>>>>>>>>> 1. If we are using the Zookeeper API for initializing the
>>>>>>>>> Zookeeper from the Helix app (out app that uses Helix) would we have
>>>>>>>>> specific process for Zookeeper? Would we have one process for our app and
>>>>>>>>> another for the Zookeeper?
>>>>>>>>> What happens if we'll not initialize the Zookeeper from the Helix
>>>>>>>>> app programmatically but using the commands, whether then would we have
>>>>>>>>> specific process for Zookeeper?
>>>>>>>>>
>>>>>>>>> 2. If we'll decide to initialize Zookeeper from Helix app,
>>>>>>>>> programmatically, How can we use Zookeeper clustering (ensemble) , is there
>>>>>>>>> any API for Zookeeper clustering, any direction would highly appreciated.
>>>>>>>>>
>>>>>>>>> 3. Is it possible to start the Zookeeper by using commands and not
>>>>>>>>> initializing specific process for it but using the Helix app (our app that
>>>>>>>>> uses Helix API)?
>>>>>>>>>
>>>>>>>>> I hope my questions are enough clear.
>>>>>>>>>
>>>>>>>>> Thanks in advance,
>>>>>>>>> Erwin Karbasi
>>>>>>>>> AT&T, Senior Software Architect
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by kishore g <g....@gmail.com>.
Hi Erwin,
Here are the options:
1. participant controller and zookeeper on one process on all 50 boxes:
This is not possible, you need only 3 or 5 zookeepers running and all
participant/controllers need to know the machinename: port of the zookeeper
boxes upfront.
2. separate zookeeper but participant and controller in one process: This
is possible, controller can be started as part of all participants but only
one controller will be active. This is fine until you have like 100 nodes,
after which its better to have a separate controller.
3. zookeeper and controller on same box but separate process, participants
in different: This is possible, note that once you this configuration all
additional clusters you deploy can share the same zookeeper/controller. So
if you plan to have many clusters this is a better option and much easier
to manage the clusters. You can use standard commands to start zookeeper
and helix controller.
4. zookeeper and controller on same box and in one process: This also works
and the advantages of 3 holds good here as well. You will have to start
both of them programmatically. Any time you restart a process, both
zookeeper and controller will be upgraded.
My preference is 3) 4) 2), you cannot do 1).
I agree with you that it requires additional monitoring but actual
experience suggests that co-locating zookeeper with participants causes
un-wanted problems. For example, the participants are heavily loaded and
your application might have large GC pauses/consume al cpu/memory on that
box. This will adversely impact the functionality of zookeeper.
Thanks,
Kishore G
On Fri, Jun 14, 2013 at 5:44 AM, Erwin Karbasi <er...@optinity.com> wrote:
> My question is, whether is it possible to collocate participant,
> controller and zookeeper on one process, is there any limitations in term
> of Helix or Zookeeper?
> Any direction regarding above approach would be appreciated.
> On Jun 14, 2013 3:23 PM, "Ming Fang" <mi...@mac.com> wrote:
>
>> That's not possible at this time.
>> ZAC currently must run as a standalone process and only allow one
>> instance.
>>
>> Sent from my iPad
>>
>> On Jun 14, 2013, at 7:35 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>
>> If I have 40 app server instances (JVMs) could I run participants on all
>> of them, controller and zookeeper just on 5 of then, but on the same
>> process of the 5 participants?
>> On Jun 14, 2013 2:29 PM, "Ming Fang" <mi...@mac.com> wrote:
>>
>>> ZAC currently runs Zookeeper in standalone mode.
>>> Therefore you can only run one instance.
>>> If you also run the participant in it then that limits you to one
>>> instance of the participant also.
>>>
>>>
>>> Sent from my iPad
>>>
>>> On Jun 14, 2013, at 7:23 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>>
>>> Hi Ming,
>>>
>>> Thank you for fast turnaround response.
>>>
>>> Could I run my app that uses Helix (participant), controller and
>>> zookeeper on the same process?
>>>
>>> Cheers,
>>> Erwin
>>> On Jun 14, 2013 2:16 PM, "Ming Fang" <mi...@mac.com> wrote:
>>>
>>>> Erwin
>>>>
>>>> Here is an example of running Zookeeper, and Controller in the same
>>>> process,
>>>>
>>>> https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java
>>>> We have this in production.
>>>>
>>>> Sent from my iPad
>>>>
>>>> On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>>>
>>>> I know that it's better that participants, controller and zookeeper
>>>> will have separate java processes but we have more then 40 boxes with app
>>>> server that we'd like to manage them with Helix Leader election and cannot
>>>> add more processes because each additional java process requires management
>>>> (monitoring, alarms, KPI, etc...), so we prefer to avoid the additional
>>>> java processes. We'll not manage to approve additional processes
>>>>
>>>> Your insight would appreciated.
>>>>
>>>> Thanks,
>>>> Erwin
>>>> On Jun 14, 2013 9:58 AM, "kishore g" <g....@gmail.com> wrote:
>>>>
>>>>> Hi Erwin,
>>>>>
>>>>> This is how I would setup your cluster. Allocate 3 boxes that only run
>>>>> zookeeper and Helix controller. Its better that they are two separate
>>>>> process. These boxes need not be powerful but isolating zookeeper and helix
>>>>> controller from actual server is a good deployment practice. See
>>>>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetupon how to use zookeeper clustering ensemble.
>>>>>
>>>>>
>>>>> Run the real servers on the rest of the boxes. Do not start zookeeper
>>>>> as part of these processes, instead simply provide the zookeeper address
>>>>> (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Kishore G
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <er...@optinity.com>wrote:
>>>>>
>>>>>> Hi Kishore,
>>>>>>
>>>>>> We have more then 50 nodes.
>>>>>>
>>>>>> Cheers,
>>>>>> Erwin
>>>>>> On Jun 14, 2013 1:51 AM, "kishore g" <g....@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Erwin,
>>>>>>>
>>>>>>> How many nodes do you have in your cluster ?
>>>>>>>
>>>>>>> thanks,
>>>>>>> Kishore G
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <er...@optinity.com>wrote:
>>>>>>>
>>>>>>>> Hello Maters,
>>>>>>>>
>>>>>>>> We don't want to have separate/specific JAVA process for Zookeeper.
>>>>>>>> As I figured out from the Master-Slave sample we can start the
>>>>>>>> Zookeeper programmatically using the Zookeeper API.
>>>>>>>>
>>>>>>>> I have several questions:
>>>>>>>>
>>>>>>>> 1. If we are using the Zookeeper API for initializing the Zookeeper
>>>>>>>> from the Helix app (out app that uses Helix) would we have specific process
>>>>>>>> for Zookeeper? Would we have one process for our app and another for the
>>>>>>>> Zookeeper?
>>>>>>>> What happens if we'll not initialize the Zookeeper from the Helix
>>>>>>>> app programmatically but using the commands, whether then would we have
>>>>>>>> specific process for Zookeeper?
>>>>>>>>
>>>>>>>> 2. If we'll decide to initialize Zookeeper from Helix app,
>>>>>>>> programmatically, How can we use Zookeeper clustering (ensemble) , is there
>>>>>>>> any API for Zookeeper clustering, any direction would highly appreciated.
>>>>>>>>
>>>>>>>> 3. Is it possible to start the Zookeeper by using commands and not
>>>>>>>> initializing specific process for it but using the Helix app (our app that
>>>>>>>> uses Helix API)?
>>>>>>>>
>>>>>>>> I hope my questions are enough clear.
>>>>>>>>
>>>>>>>> Thanks in advance,
>>>>>>>> Erwin Karbasi
>>>>>>>> AT&T, Senior Software Architect
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by Erwin Karbasi <er...@optinity.com>.
My question is, whether is it possible to collocate participant,
controller and zookeeper on one process, is there any limitations in term
of Helix or Zookeeper?
Any direction regarding above approach would be appreciated.
On Jun 14, 2013 3:23 PM, "Ming Fang" <mi...@mac.com> wrote:
> That's not possible at this time.
> ZAC currently must run as a standalone process and only allow one
> instance.
>
> Sent from my iPad
>
> On Jun 14, 2013, at 7:35 AM, Erwin Karbasi <er...@optinity.com> wrote:
>
> If I have 40 app server instances (JVMs) could I run participants on all
> of them, controller and zookeeper just on 5 of then, but on the same
> process of the 5 participants?
> On Jun 14, 2013 2:29 PM, "Ming Fang" <mi...@mac.com> wrote:
>
>> ZAC currently runs Zookeeper in standalone mode.
>> Therefore you can only run one instance.
>> If you also run the participant in it then that limits you to one
>> instance of the participant also.
>>
>>
>> Sent from my iPad
>>
>> On Jun 14, 2013, at 7:23 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>
>> Hi Ming,
>>
>> Thank you for fast turnaround response.
>>
>> Could I run my app that uses Helix (participant), controller and
>> zookeeper on the same process?
>>
>> Cheers,
>> Erwin
>> On Jun 14, 2013 2:16 PM, "Ming Fang" <mi...@mac.com> wrote:
>>
>>> Erwin
>>>
>>> Here is an example of running Zookeeper, and Controller in the same
>>> process,
>>>
>>> https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java
>>> We have this in production.
>>>
>>> Sent from my iPad
>>>
>>> On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>>
>>> I know that it's better that participants, controller and zookeeper will
>>> have separate java processes but we have more then 40 boxes with app server
>>> that we'd like to manage them with Helix Leader election and cannot add
>>> more processes because each additional java process requires management
>>> (monitoring, alarms, KPI, etc...), so we prefer to avoid the additional
>>> java processes. We'll not manage to approve additional processes
>>>
>>> Your insight would appreciated.
>>>
>>> Thanks,
>>> Erwin
>>> On Jun 14, 2013 9:58 AM, "kishore g" <g....@gmail.com> wrote:
>>>
>>>> Hi Erwin,
>>>>
>>>> This is how I would setup your cluster. Allocate 3 boxes that only run
>>>> zookeeper and Helix controller. Its better that they are two separate
>>>> process. These boxes need not be powerful but isolating zookeeper and helix
>>>> controller from actual server is a good deployment practice. See
>>>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetupon how to use zookeeper clustering ensemble.
>>>>
>>>>
>>>> Run the real servers on the rest of the boxes. Do not start zookeeper
>>>> as part of these processes, instead simply provide the zookeeper address
>>>> (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api.
>>>>
>>>>
>>>> Thanks,
>>>> Kishore G
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <er...@optinity.com>wrote:
>>>>
>>>>> Hi Kishore,
>>>>>
>>>>> We have more then 50 nodes.
>>>>>
>>>>> Cheers,
>>>>> Erwin
>>>>> On Jun 14, 2013 1:51 AM, "kishore g" <g....@gmail.com> wrote:
>>>>>
>>>>>> Hi Erwin,
>>>>>>
>>>>>> How many nodes do you have in your cluster ?
>>>>>>
>>>>>> thanks,
>>>>>> Kishore G
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <er...@optinity.com>wrote:
>>>>>>
>>>>>>> Hello Maters,
>>>>>>>
>>>>>>> We don't want to have separate/specific JAVA process for Zookeeper.
>>>>>>> As I figured out from the Master-Slave sample we can start the
>>>>>>> Zookeeper programmatically using the Zookeeper API.
>>>>>>>
>>>>>>> I have several questions:
>>>>>>>
>>>>>>> 1. If we are using the Zookeeper API for initializing the Zookeeper
>>>>>>> from the Helix app (out app that uses Helix) would we have specific process
>>>>>>> for Zookeeper? Would we have one process for our app and another for the
>>>>>>> Zookeeper?
>>>>>>> What happens if we'll not initialize the Zookeeper from the Helix
>>>>>>> app programmatically but using the commands, whether then would we have
>>>>>>> specific process for Zookeeper?
>>>>>>>
>>>>>>> 2. If we'll decide to initialize Zookeeper from Helix app,
>>>>>>> programmatically, How can we use Zookeeper clustering (ensemble) , is there
>>>>>>> any API for Zookeeper clustering, any direction would highly appreciated.
>>>>>>>
>>>>>>> 3. Is it possible to start the Zookeeper by using commands and not
>>>>>>> initializing specific process for it but using the Helix app (our app that
>>>>>>> uses Helix API)?
>>>>>>>
>>>>>>> I hope my questions are enough clear.
>>>>>>>
>>>>>>> Thanks in advance,
>>>>>>> Erwin Karbasi
>>>>>>> AT&T, Senior Software Architect
>>>>>>>
>>>>>>
>>>>>>
>>>>
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by Ming Fang <mi...@mac.com>.
That's not possible at this time.
ZAC currently must run as a standalone process and only allow one instance.
Sent from my iPad
On Jun 14, 2013, at 7:35 AM, Erwin Karbasi <er...@optinity.com> wrote:
> If I have 40 app server instances (JVMs) could I run participants on all of them, controller and zookeeper just on 5 of then, but on the same process of the 5 participants?
>
> On Jun 14, 2013 2:29 PM, "Ming Fang" <mi...@mac.com> wrote:
>> ZAC currently runs Zookeeper in standalone mode.
>> Therefore you can only run one instance.
>> If you also run the participant in it then that limits you to one instance of the participant also.
>>
>>
>> Sent from my iPad
>>
>> On Jun 14, 2013, at 7:23 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>
>>> Hi Ming,
>>>
>>> Thank you for fast turnaround response.
>>>
>>> Could I run my app that uses Helix (participant), controller and zookeeper on the same process?
>>>
>>> Cheers,
>>> Erwin
>>>
>>> On Jun 14, 2013 2:16 PM, "Ming Fang" <mi...@mac.com> wrote:
>>>> Erwin
>>>>
>>>> Here is an example of running Zookeeper, and Controller in the same process,
>>>> https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java
>>>> We have this in production.
>>>>
>>>> Sent from my iPad
>>>>
>>>> On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>>>
>>>>> I know that it's better that participants, controller and zookeeper will have separate java processes but we have more then 40 boxes with app server that we'd like to manage them with Helix Leader election and cannot add more processes because each additional java process requires management (monitoring, alarms, KPI, etc...), so we prefer to avoid the additional java processes. We'll not manage to approve additional processes
>>>>>
>>>>> Your insight would appreciated.
>>>>>
>>>>> Thanks,
>>>>> Erwin
>>>>>
>>>>> On Jun 14, 2013 9:58 AM, "kishore g" <g....@gmail.com> wrote:
>>>>>> Hi Erwin,
>>>>>>
>>>>>> This is how I would setup your cluster. Allocate 3 boxes that only run zookeeper and Helix controller. Its better that they are two separate process. These boxes need not be powerful but isolating zookeeper and helix controller from actual server is a good deployment practice. See http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup on how to use zookeeper clustering ensemble.
>>>>>>
>>>>>>
>>>>>> Run the real servers on the rest of the boxes. Do not start zookeeper as part of these processes, instead simply provide the zookeeper address (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore G
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <er...@optinity.com> wrote:
>>>>>>> Hi Kishore,
>>>>>>>
>>>>>>> We have more then 50 nodes.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Erwin
>>>>>>>
>>>>>>> On Jun 14, 2013 1:51 AM, "kishore g" <g....@gmail.com> wrote:
>>>>>>>> Hi Erwin,
>>>>>>>>
>>>>>>>> How many nodes do you have in your cluster ?
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Kishore G
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <er...@optinity.com> wrote:
>>>>>>>>> Hello Maters,
>>>>>>>>>
>>>>>>>>> We don't want to have separate/specific JAVA process for Zookeeper.
>>>>>>>>> As I figured out from the Master-Slave sample we can start the Zookeeper programmatically using the Zookeeper API.
>>>>>>>>>
>>>>>>>>> I have several questions:
>>>>>>>>>
>>>>>>>>> 1. If we are using the Zookeeper API for initializing the Zookeeper from the Helix app (out app that uses Helix) would we have specific process for Zookeeper? Would we have one process for our app and another for the Zookeeper?
>>>>>>>>> What happens if we'll not initialize the Zookeeper from the Helix app programmatically but using the commands, whether then would we have specific process for Zookeeper?
>>>>>>>>>
>>>>>>>>> 2. If we'll decide to initialize Zookeeper from Helix app, programmatically, How can we use Zookeeper clustering (ensemble) , is there any API for Zookeeper clustering, any direction would highly appreciated.
>>>>>>>>>
>>>>>>>>> 3. Is it possible to start the Zookeeper by using commands and not initializing specific process for it but using the Helix app (our app that uses Helix API)?
>>>>>>>>>
>>>>>>>>> I hope my questions are enough clear.
>>>>>>>>>
>>>>>>>>> Thanks in advance,
>>>>>>>>> Erwin Karbasi
>>>>>>>>> AT&T, Senior Software Architect
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by Erwin Karbasi <er...@optinity.com>.
If I have 40 app server instances (JVMs) could I run participants on all of
them, controller and zookeeper just on 5 of then, but on the same process
of the 5 participants?
On Jun 14, 2013 2:29 PM, "Ming Fang" <mi...@mac.com> wrote:
> ZAC currently runs Zookeeper in standalone mode.
> Therefore you can only run one instance.
> If you also run the participant in it then that limits you to one instance
> of the participant also.
>
>
> Sent from my iPad
>
> On Jun 14, 2013, at 7:23 AM, Erwin Karbasi <er...@optinity.com> wrote:
>
> Hi Ming,
>
> Thank you for fast turnaround response.
>
> Could I run my app that uses Helix (participant), controller and zookeeper
> on the same process?
>
> Cheers,
> Erwin
> On Jun 14, 2013 2:16 PM, "Ming Fang" <mi...@mac.com> wrote:
>
>> Erwin
>>
>> Here is an example of running Zookeeper, and Controller in the same
>> process,
>>
>> https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java
>> We have this in production.
>>
>> Sent from my iPad
>>
>> On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>
>> I know that it's better that participants, controller and zookeeper will
>> have separate java processes but we have more then 40 boxes with app server
>> that we'd like to manage them with Helix Leader election and cannot add
>> more processes because each additional java process requires management
>> (monitoring, alarms, KPI, etc...), so we prefer to avoid the additional
>> java processes. We'll not manage to approve additional processes
>>
>> Your insight would appreciated.
>>
>> Thanks,
>> Erwin
>> On Jun 14, 2013 9:58 AM, "kishore g" <g....@gmail.com> wrote:
>>
>>> Hi Erwin,
>>>
>>> This is how I would setup your cluster. Allocate 3 boxes that only run
>>> zookeeper and Helix controller. Its better that they are two separate
>>> process. These boxes need not be powerful but isolating zookeeper and helix
>>> controller from actual server is a good deployment practice. See
>>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetupon how to use zookeeper clustering ensemble.
>>>
>>>
>>> Run the real servers on the rest of the boxes. Do not start zookeeper as
>>> part of these processes, instead simply provide the zookeeper address
>>> (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api.
>>>
>>>
>>> Thanks,
>>> Kishore G
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <er...@optinity.com>wrote:
>>>
>>>> Hi Kishore,
>>>>
>>>> We have more then 50 nodes.
>>>>
>>>> Cheers,
>>>> Erwin
>>>> On Jun 14, 2013 1:51 AM, "kishore g" <g....@gmail.com> wrote:
>>>>
>>>>> Hi Erwin,
>>>>>
>>>>> How many nodes do you have in your cluster ?
>>>>>
>>>>> thanks,
>>>>> Kishore G
>>>>>
>>>>>
>>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <er...@optinity.com>wrote:
>>>>>
>>>>>> Hello Maters,
>>>>>>
>>>>>> We don't want to have separate/specific JAVA process for Zookeeper.
>>>>>> As I figured out from the Master-Slave sample we can start the
>>>>>> Zookeeper programmatically using the Zookeeper API.
>>>>>>
>>>>>> I have several questions:
>>>>>>
>>>>>> 1. If we are using the Zookeeper API for initializing the Zookeeper
>>>>>> from the Helix app (out app that uses Helix) would we have specific process
>>>>>> for Zookeeper? Would we have one process for our app and another for the
>>>>>> Zookeeper?
>>>>>> What happens if we'll not initialize the Zookeeper from the Helix app
>>>>>> programmatically but using the commands, whether then would we have
>>>>>> specific process for Zookeeper?
>>>>>>
>>>>>> 2. If we'll decide to initialize Zookeeper from Helix app,
>>>>>> programmatically, How can we use Zookeeper clustering (ensemble) , is there
>>>>>> any API for Zookeeper clustering, any direction would highly appreciated.
>>>>>>
>>>>>> 3. Is it possible to start the Zookeeper by using commands and not
>>>>>> initializing specific process for it but using the Helix app (our app that
>>>>>> uses Helix API)?
>>>>>>
>>>>>> I hope my questions are enough clear.
>>>>>>
>>>>>> Thanks in advance,
>>>>>> Erwin Karbasi
>>>>>> AT&T, Senior Software Architect
>>>>>>
>>>>>
>>>>>
>>>
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by Ming Fang <mi...@mac.com>.
ZAC currently runs Zookeeper in standalone mode.
Therefore you can only run one instance.
If you also run the participant in it then that limits you to one instance of the participant also.
Sent from my iPad
On Jun 14, 2013, at 7:23 AM, Erwin Karbasi <er...@optinity.com> wrote:
> Hi Ming,
>
> Thank you for fast turnaround response.
>
> Could I run my app that uses Helix (participant), controller and zookeeper on the same process?
>
> Cheers,
> Erwin
>
> On Jun 14, 2013 2:16 PM, "Ming Fang" <mi...@mac.com> wrote:
>> Erwin
>>
>> Here is an example of running Zookeeper, and Controller in the same process,
>> https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java
>> We have this in production.
>>
>> Sent from my iPad
>>
>> On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <er...@optinity.com> wrote:
>>
>>> I know that it's better that participants, controller and zookeeper will have separate java processes but we have more then 40 boxes with app server that we'd like to manage them with Helix Leader election and cannot add more processes because each additional java process requires management (monitoring, alarms, KPI, etc...), so we prefer to avoid the additional java processes. We'll not manage to approve additional processes
>>>
>>> Your insight would appreciated.
>>>
>>> Thanks,
>>> Erwin
>>>
>>> On Jun 14, 2013 9:58 AM, "kishore g" <g....@gmail.com> wrote:
>>>> Hi Erwin,
>>>>
>>>> This is how I would setup your cluster. Allocate 3 boxes that only run zookeeper and Helix controller. Its better that they are two separate process. These boxes need not be powerful but isolating zookeeper and helix controller from actual server is a good deployment practice. See http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup on how to use zookeeper clustering ensemble.
>>>>
>>>>
>>>> Run the real servers on the rest of the boxes. Do not start zookeeper as part of these processes, instead simply provide the zookeeper address (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api.
>>>>
>>>>
>>>> Thanks,
>>>> Kishore G
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <er...@optinity.com> wrote:
>>>>> Hi Kishore,
>>>>>
>>>>> We have more then 50 nodes.
>>>>>
>>>>> Cheers,
>>>>> Erwin
>>>>>
>>>>> On Jun 14, 2013 1:51 AM, "kishore g" <g....@gmail.com> wrote:
>>>>>> Hi Erwin,
>>>>>>
>>>>>> How many nodes do you have in your cluster ?
>>>>>>
>>>>>> thanks,
>>>>>> Kishore G
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <er...@optinity.com> wrote:
>>>>>>> Hello Maters,
>>>>>>>
>>>>>>> We don't want to have separate/specific JAVA process for Zookeeper.
>>>>>>> As I figured out from the Master-Slave sample we can start the Zookeeper programmatically using the Zookeeper API.
>>>>>>>
>>>>>>> I have several questions:
>>>>>>>
>>>>>>> 1. If we are using the Zookeeper API for initializing the Zookeeper from the Helix app (out app that uses Helix) would we have specific process for Zookeeper? Would we have one process for our app and another for the Zookeeper?
>>>>>>> What happens if we'll not initialize the Zookeeper from the Helix app programmatically but using the commands, whether then would we have specific process for Zookeeper?
>>>>>>>
>>>>>>> 2. If we'll decide to initialize Zookeeper from Helix app, programmatically, How can we use Zookeeper clustering (ensemble) , is there any API for Zookeeper clustering, any direction would highly appreciated.
>>>>>>>
>>>>>>> 3. Is it possible to start the Zookeeper by using commands and not initializing specific process for it but using the Helix app (our app that uses Helix API)?
>>>>>>>
>>>>>>> I hope my questions are enough clear.
>>>>>>>
>>>>>>> Thanks in advance,
>>>>>>> Erwin Karbasi
>>>>>>> AT&T, Senior Software Architect
>>>>>>
>>>>
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by Erwin Karbasi <er...@optinity.com>.
Hi Ming,
Thank you for fast turnaround response.
Could I run my app that uses Helix (participant), controller and zookeeper
on the same process?
Cheers,
Erwin
On Jun 14, 2013 2:16 PM, "Ming Fang" <mi...@mac.com> wrote:
> Erwin
>
> Here is an example of running Zookeeper, and Controller in the same
> process,
>
> https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java
> We have this in production.
>
> Sent from my iPad
>
> On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <er...@optinity.com> wrote:
>
> I know that it's better that participants, controller and zookeeper will
> have separate java processes but we have more then 40 boxes with app server
> that we'd like to manage them with Helix Leader election and cannot add
> more processes because each additional java process requires management
> (monitoring, alarms, KPI, etc...), so we prefer to avoid the additional
> java processes. We'll not manage to approve additional processes
>
> Your insight would appreciated.
>
> Thanks,
> Erwin
> On Jun 14, 2013 9:58 AM, "kishore g" <g....@gmail.com> wrote:
>
>> Hi Erwin,
>>
>> This is how I would setup your cluster. Allocate 3 boxes that only run
>> zookeeper and Helix controller. Its better that they are two separate
>> process. These boxes need not be powerful but isolating zookeeper and helix
>> controller from actual server is a good deployment practice. See
>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetupon how to use zookeeper clustering ensemble.
>>
>>
>> Run the real servers on the rest of the boxes. Do not start zookeeper as
>> part of these processes, instead simply provide the zookeeper address
>> (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api.
>>
>>
>> Thanks,
>> Kishore G
>>
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <er...@optinity.com>wrote:
>>
>>> Hi Kishore,
>>>
>>> We have more then 50 nodes.
>>>
>>> Cheers,
>>> Erwin
>>> On Jun 14, 2013 1:51 AM, "kishore g" <g....@gmail.com> wrote:
>>>
>>>> Hi Erwin,
>>>>
>>>> How many nodes do you have in your cluster ?
>>>>
>>>> thanks,
>>>> Kishore G
>>>>
>>>>
>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <er...@optinity.com>wrote:
>>>>
>>>>> Hello Maters,
>>>>>
>>>>> We don't want to have separate/specific JAVA process for Zookeeper.
>>>>> As I figured out from the Master-Slave sample we can start the
>>>>> Zookeeper programmatically using the Zookeeper API.
>>>>>
>>>>> I have several questions:
>>>>>
>>>>> 1. If we are using the Zookeeper API for initializing the Zookeeper
>>>>> from the Helix app (out app that uses Helix) would we have specific process
>>>>> for Zookeeper? Would we have one process for our app and another for the
>>>>> Zookeeper?
>>>>> What happens if we'll not initialize the Zookeeper from the Helix app
>>>>> programmatically but using the commands, whether then would we have
>>>>> specific process for Zookeeper?
>>>>>
>>>>> 2. If we'll decide to initialize Zookeeper from Helix app,
>>>>> programmatically, How can we use Zookeeper clustering (ensemble) , is there
>>>>> any API for Zookeeper clustering, any direction would highly appreciated.
>>>>>
>>>>> 3. Is it possible to start the Zookeeper by using commands and not
>>>>> initializing specific process for it but using the Helix app (our app that
>>>>> uses Helix API)?
>>>>>
>>>>> I hope my questions are enough clear.
>>>>>
>>>>> Thanks in advance,
>>>>> Erwin Karbasi
>>>>> AT&T, Senior Software Architect
>>>>>
>>>>
>>>>
>>
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by Ming Fang <mi...@mac.com>.
Erwin
Here is an example of running Zookeeper, and Controller in the same process,
https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java
We have this in production.
Sent from my iPad
On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <er...@optinity.com> wrote:
> I know that it's better that participants, controller and zookeeper will have separate java processes but we have more then 40 boxes with app server that we'd like to manage them with Helix Leader election and cannot add more processes because each additional java process requires management (monitoring, alarms, KPI, etc...), so we prefer to avoid the additional java processes. We'll not manage to approve additional processes
>
> Your insight would appreciated.
>
> Thanks,
> Erwin
>
> On Jun 14, 2013 9:58 AM, "kishore g" <g....@gmail.com> wrote:
>> Hi Erwin,
>>
>> This is how I would setup your cluster. Allocate 3 boxes that only run zookeeper and Helix controller. Its better that they are two separate process. These boxes need not be powerful but isolating zookeeper and helix controller from actual server is a good deployment practice. See http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup on how to use zookeeper clustering ensemble.
>>
>>
>> Run the real servers on the rest of the boxes. Do not start zookeeper as part of these processes, instead simply provide the zookeeper address (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api.
>>
>>
>> Thanks,
>> Kishore G
>>
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <er...@optinity.com> wrote:
>>> Hi Kishore,
>>>
>>> We have more then 50 nodes.
>>>
>>> Cheers,
>>> Erwin
>>>
>>> On Jun 14, 2013 1:51 AM, "kishore g" <g....@gmail.com> wrote:
>>>> Hi Erwin,
>>>>
>>>> How many nodes do you have in your cluster ?
>>>>
>>>> thanks,
>>>> Kishore G
>>>>
>>>>
>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <er...@optinity.com> wrote:
>>>>> Hello Maters,
>>>>>
>>>>> We don't want to have separate/specific JAVA process for Zookeeper.
>>>>> As I figured out from the Master-Slave sample we can start the Zookeeper programmatically using the Zookeeper API.
>>>>>
>>>>> I have several questions:
>>>>>
>>>>> 1. If we are using the Zookeeper API for initializing the Zookeeper from the Helix app (out app that uses Helix) would we have specific process for Zookeeper? Would we have one process for our app and another for the Zookeeper?
>>>>> What happens if we'll not initialize the Zookeeper from the Helix app programmatically but using the commands, whether then would we have specific process for Zookeeper?
>>>>>
>>>>> 2. If we'll decide to initialize Zookeeper from Helix app, programmatically, How can we use Zookeeper clustering (ensemble) , is there any API for Zookeeper clustering, any direction would highly appreciated.
>>>>>
>>>>> 3. Is it possible to start the Zookeeper by using commands and not initializing specific process for it but using the Helix app (our app that uses Helix API)?
>>>>>
>>>>> I hope my questions are enough clear.
>>>>>
>>>>> Thanks in advance,
>>>>> Erwin Karbasi
>>>>> AT&T, Senior Software Architect
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by Erwin Karbasi <er...@optinity.com>.
I know that it's better that participants, controller and zookeeper will
have separate java processes but we have more then 40 boxes with app server
that we'd like to manage them with Helix Leader election and cannot add
more processes because each additional java process requires management
(monitoring, alarms, KPI, etc...), so we prefer to avoid the additional
java processes. We'll not manage to approve additional processes
Your insight would appreciated.
Thanks,
Erwin
On Jun 14, 2013 9:58 AM, "kishore g" <g....@gmail.com> wrote:
> Hi Erwin,
>
> This is how I would setup your cluster. Allocate 3 boxes that only run
> zookeeper and Helix controller. Its better that they are two separate
> process. These boxes need not be powerful but isolating zookeeper and helix
> controller from actual server is a good deployment practice. See
> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetupon how to use zookeeper clustering ensemble.
>
>
> Run the real servers on the rest of the boxes. Do not start zookeeper as
> part of these processes, instead simply provide the zookeeper address
> (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api.
>
>
> Thanks,
> Kishore G
>
>
>
>
>
>
>
>
> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <er...@optinity.com>wrote:
>
>> Hi Kishore,
>>
>> We have more then 50 nodes.
>>
>> Cheers,
>> Erwin
>> On Jun 14, 2013 1:51 AM, "kishore g" <g....@gmail.com> wrote:
>>
>>> Hi Erwin,
>>>
>>> How many nodes do you have in your cluster ?
>>>
>>> thanks,
>>> Kishore G
>>>
>>>
>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <er...@optinity.com>wrote:
>>>
>>>> Hello Maters,
>>>>
>>>> We don't want to have separate/specific JAVA process for Zookeeper.
>>>> As I figured out from the Master-Slave sample we can start the
>>>> Zookeeper programmatically using the Zookeeper API.
>>>>
>>>> I have several questions:
>>>>
>>>> 1. If we are using the Zookeeper API for initializing the Zookeeper
>>>> from the Helix app (out app that uses Helix) would we have specific process
>>>> for Zookeeper? Would we have one process for our app and another for the
>>>> Zookeeper?
>>>> What happens if we'll not initialize the Zookeeper from the Helix app
>>>> programmatically but using the commands, whether then would we have
>>>> specific process for Zookeeper?
>>>>
>>>> 2. If we'll decide to initialize Zookeeper from Helix app,
>>>> programmatically, How can we use Zookeeper clustering (ensemble) , is there
>>>> any API for Zookeeper clustering, any direction would highly appreciated.
>>>>
>>>> 3. Is it possible to start the Zookeeper by using commands and not
>>>> initializing specific process for it but using the Helix app (our app that
>>>> uses Helix API)?
>>>>
>>>> I hope my questions are enough clear.
>>>>
>>>> Thanks in advance,
>>>> Erwin Karbasi
>>>> AT&T, Senior Software Architect
>>>>
>>>
>>>
>
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by kishore g <g....@gmail.com>.
Hi Erwin,
This is how I would setup your cluster. Allocate 3 boxes that only run
zookeeper and Helix controller. Its better that they are two separate
process. These boxes need not be powerful but isolating zookeeper and helix
controller from actual server is a good deployment practice. See
http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetupon
how to use zookeeper clustering ensemble.
Run the real servers on the rest of the boxes. Do not start zookeeper as
part of these processes, instead simply provide the zookeeper address
(zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api.
Thanks,
Kishore G
On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <er...@optinity.com> wrote:
> Hi Kishore,
>
> We have more then 50 nodes.
>
> Cheers,
> Erwin
> On Jun 14, 2013 1:51 AM, "kishore g" <g....@gmail.com> wrote:
>
>> Hi Erwin,
>>
>> How many nodes do you have in your cluster ?
>>
>> thanks,
>> Kishore G
>>
>>
>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <er...@optinity.com>wrote:
>>
>>> Hello Maters,
>>>
>>> We don't want to have separate/specific JAVA process for Zookeeper.
>>> As I figured out from the Master-Slave sample we can start the Zookeeper
>>> programmatically using the Zookeeper API.
>>>
>>> I have several questions:
>>>
>>> 1. If we are using the Zookeeper API for initializing the Zookeeper from
>>> the Helix app (out app that uses Helix) would we have specific process for
>>> Zookeeper? Would we have one process for our app and another for the
>>> Zookeeper?
>>> What happens if we'll not initialize the Zookeeper from the Helix app
>>> programmatically but using the commands, whether then would we have
>>> specific process for Zookeeper?
>>>
>>> 2. If we'll decide to initialize Zookeeper from Helix app,
>>> programmatically, How can we use Zookeeper clustering (ensemble) , is there
>>> any API for Zookeeper clustering, any direction would highly appreciated.
>>>
>>> 3. Is it possible to start the Zookeeper by using commands and not
>>> initializing specific process for it but using the Helix app (our app that
>>> uses Helix API)?
>>>
>>> I hope my questions are enough clear.
>>>
>>> Thanks in advance,
>>> Erwin Karbasi
>>> AT&T, Senior Software Architect
>>>
>>
>>
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by Erwin Karbasi <er...@optinity.com>.
Hi Kishore,
We have more then 50 nodes.
Cheers,
Erwin
On Jun 14, 2013 1:51 AM, "kishore g" <g....@gmail.com> wrote:
> Hi Erwin,
>
> How many nodes do you have in your cluster ?
>
> thanks,
> Kishore G
>
>
> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <er...@optinity.com> wrote:
>
>> Hello Maters,
>>
>> We don't want to have separate/specific JAVA process for Zookeeper.
>> As I figured out from the Master-Slave sample we can start the Zookeeper
>> programmatically using the Zookeeper API.
>>
>> I have several questions:
>>
>> 1. If we are using the Zookeeper API for initializing the Zookeeper from
>> the Helix app (out app that uses Helix) would we have specific process for
>> Zookeeper? Would we have one process for our app and another for the
>> Zookeeper?
>> What happens if we'll not initialize the Zookeeper from the Helix app
>> programmatically but using the commands, whether then would we have
>> specific process for Zookeeper?
>>
>> 2. If we'll decide to initialize Zookeeper from Helix app,
>> programmatically, How can we use Zookeeper clustering (ensemble) , is there
>> any API for Zookeeper clustering, any direction would highly appreciated.
>>
>> 3. Is it possible to start the Zookeeper by using commands and not
>> initializing specific process for it but using the Helix app (our app that
>> uses Helix API)?
>>
>> I hope my questions are enough clear.
>>
>> Thanks in advance,
>> Erwin Karbasi
>> AT&T, Senior Software Architect
>>
>
>
Re: Is zookeeper for Helix has standalone JAVA process?
Posted by kishore g <g....@gmail.com>.
Hi Erwin,
How many nodes do you have in your cluster ?
thanks,
Kishore G
On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <er...@optinity.com> wrote:
> Hello Maters,
>
> We don't want to have separate/specific JAVA process for Zookeeper.
> As I figured out from the Master-Slave sample we can start the Zookeeper
> programmatically using the Zookeeper API.
>
> I have several questions:
>
> 1. If we are using the Zookeeper API for initializing the Zookeeper from
> the Helix app (out app that uses Helix) would we have specific process for
> Zookeeper? Would we have one process for our app and another for the
> Zookeeper?
> What happens if we'll not initialize the Zookeeper from the Helix app
> programmatically but using the commands, whether then would we have
> specific process for Zookeeper?
>
> 2. If we'll decide to initialize Zookeeper from Helix app,
> programmatically, How can we use Zookeeper clustering (ensemble) , is there
> any API for Zookeeper clustering, any direction would highly appreciated.
>
> 3. Is it possible to start the Zookeeper by using commands and not
> initializing specific process for it but using the Helix app (our app that
> uses Helix API)?
>
> I hope my questions are enough clear.
>
> Thanks in advance,
> Erwin Karbasi
> AT&T, Senior Software Architect
>