You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Aniket Chakrabarti <ch...@cse.ohio-state.edu> on 2011/12/15 05:44:15 UTC
Running zookeeper on multiple EC2 instances
Hi,
I need some suggestion on how to automate zookeeper deployment on EC2
instances. I know how to run zookeeper on a single EC2 instance
manually. But I want to automate such that every time a new instance is
spawned, zookeeper will be run on it. There will be around 200 such
instances.
Any pointers will be helpful.
Thanks,
Aniket
Re: Running zookeeper on multiple EC2 instances
Posted by Andrei Savu <sa...@gmail.com>.
On Thu, Dec 15, 2011 at 8:54 AM, Aniket Chakrabarti
<ch...@cse.ohio-state.edu> wrote:
> I'm testing the latency with different ratio of read/write request mix with
> different sized ensemble. 15 is the max ensemble size I will test.
Interesting experiment. Make sure you take a look at the setup scripts
in Whirr.
Also you may need to change them to optimise the deployment for performance:
* use different disks / EBS volumes for snapshots and transaction log
* OR use a tmpfs for both
Just some ideas. I would love to see the results.
Re: Running zookeeper on multiple EC2 instances
Posted by Mahadev Konar <ma...@hortonworks.com>.
Aniket,
Do keep us posted on the results. It would be interesting to learn
from your findings.
thanks
mahadev
On Wed, Dec 14, 2011 at 10:54 PM, Aniket Chakrabarti
<ch...@cse.ohio-state.edu> wrote:
> I'm testing the latency with different ratio of read/write request mix with
> different sized ensemble. 15 is the max ensemble size I will test.
>
> Aniket
>
>
> On 12/15/2011 1:35 AM, Patrick Hunt wrote:
>>
>> On Wed, Dec 14, 2011 at 10:31 PM, Aniket Chakrabarti
>> <ch...@cse.ohio-state.edu> wrote:
>>>
>>> Yes, I think whirr will work in my case.
>>>
>>> I am actually trying to create around 15 node replicated zookeeper group.
>>> And I have to create around 10 such groups.
>>>
>> Interesting. Why a 15 server ensemble?
>>
>>> So from what I understood is I basically have to run the "whirr launch
>>> cluster" command 10 times?
>>
>> Yes. That will do it.
>>
>> Patrick
>>
>>> On 12/15/2011 12:27 AM, Patrick Hunt wrote:
>>>>
>>>> Take a look at whirr. http://whirr.apache.org/
>>>>
>>>> That said, it's not clear to me what you mean. "every time a new
>>>> instance is spawned zk will run on it". Surely you don't mean a zk
>>>> server (the max you might run is say 5 or perhaps 7 servers), however
>>>> if it's a client then you'd be including the zk jar with your
>>>> application code. Perhaps you could elaborate?
>>>>
>>>> Patrick
>>>>
>>>> On Wed, Dec 14, 2011 at 8:44 PM, Aniket Chakrabarti
>>>> <ch...@cse.ohio-state.edu> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I need some suggestion on how to automate zookeeper deployment on EC2
>>>>> instances. I know how to run zookeeper on a single EC2 instance
>>>>> manually.
>>>>> But I want to automate such that every time a new instance is spawned,
>>>>> zookeeper will be run on it. There will be around 200 such instances.
>>>>>
>>>>> Any pointers will be helpful.
>>>>>
>>>>> Thanks,
>>>>> Aniket
Re: Running zookeeper on multiple EC2 instances
Posted by Aniket Chakrabarti <ch...@cse.ohio-state.edu>.
I'm testing the latency with different ratio of read/write request mix
with different sized ensemble. 15 is the max ensemble size I will test.
Aniket
On 12/15/2011 1:35 AM, Patrick Hunt wrote:
> On Wed, Dec 14, 2011 at 10:31 PM, Aniket Chakrabarti
> <ch...@cse.ohio-state.edu> wrote:
>> Yes, I think whirr will work in my case.
>>
>> I am actually trying to create around 15 node replicated zookeeper group.
>> And I have to create around 10 such groups.
>>
> Interesting. Why a 15 server ensemble?
>
>> So from what I understood is I basically have to run the "whirr launch
>> cluster" command 10 times?
> Yes. That will do it.
>
> Patrick
>
>> On 12/15/2011 12:27 AM, Patrick Hunt wrote:
>>> Take a look at whirr. http://whirr.apache.org/
>>>
>>> That said, it's not clear to me what you mean. "every time a new
>>> instance is spawned zk will run on it". Surely you don't mean a zk
>>> server (the max you might run is say 5 or perhaps 7 servers), however
>>> if it's a client then you'd be including the zk jar with your
>>> application code. Perhaps you could elaborate?
>>>
>>> Patrick
>>>
>>> On Wed, Dec 14, 2011 at 8:44 PM, Aniket Chakrabarti
>>> <ch...@cse.ohio-state.edu> wrote:
>>>> Hi,
>>>>
>>>> I need some suggestion on how to automate zookeeper deployment on EC2
>>>> instances. I know how to run zookeeper on a single EC2 instance manually.
>>>> But I want to automate such that every time a new instance is spawned,
>>>> zookeeper will be run on it. There will be around 200 such instances.
>>>>
>>>> Any pointers will be helpful.
>>>>
>>>> Thanks,
>>>> Aniket
Re: Running zookeeper on multiple EC2 instances
Posted by Patrick Hunt <ph...@apache.org>.
On Wed, Dec 14, 2011 at 10:31 PM, Aniket Chakrabarti
<ch...@cse.ohio-state.edu> wrote:
> Yes, I think whirr will work in my case.
>
> I am actually trying to create around 15 node replicated zookeeper group.
> And I have to create around 10 such groups.
>
Interesting. Why a 15 server ensemble?
> So from what I understood is I basically have to run the "whirr launch
> cluster" command 10 times?
Yes. That will do it.
Patrick
>
> On 12/15/2011 12:27 AM, Patrick Hunt wrote:
>>
>> Take a look at whirr. http://whirr.apache.org/
>>
>> That said, it's not clear to me what you mean. "every time a new
>> instance is spawned zk will run on it". Surely you don't mean a zk
>> server (the max you might run is say 5 or perhaps 7 servers), however
>> if it's a client then you'd be including the zk jar with your
>> application code. Perhaps you could elaborate?
>>
>> Patrick
>>
>> On Wed, Dec 14, 2011 at 8:44 PM, Aniket Chakrabarti
>> <ch...@cse.ohio-state.edu> wrote:
>>>
>>> Hi,
>>>
>>> I need some suggestion on how to automate zookeeper deployment on EC2
>>> instances. I know how to run zookeeper on a single EC2 instance manually.
>>> But I want to automate such that every time a new instance is spawned,
>>> zookeeper will be run on it. There will be around 200 such instances.
>>>
>>> Any pointers will be helpful.
>>>
>>> Thanks,
>>> Aniket
Re: Running zookeeper on multiple EC2 instances
Posted by Aniket Chakrabarti <ch...@cse.ohio-state.edu>.
Yes, I think whirr will work in my case.
I am actually trying to create around 15 node replicated zookeeper
group. And I have to create around 10 such groups.
So from what I understood is I basically have to run the "whirr launch
cluster" command 10 times?
Thanks,
Aniket
On 12/15/2011 12:27 AM, Patrick Hunt wrote:
> Take a look at whirr. http://whirr.apache.org/
>
> That said, it's not clear to me what you mean. "every time a new
> instance is spawned zk will run on it". Surely you don't mean a zk
> server (the max you might run is say 5 or perhaps 7 servers), however
> if it's a client then you'd be including the zk jar with your
> application code. Perhaps you could elaborate?
>
> Patrick
>
> On Wed, Dec 14, 2011 at 8:44 PM, Aniket Chakrabarti
> <ch...@cse.ohio-state.edu> wrote:
>> Hi,
>>
>> I need some suggestion on how to automate zookeeper deployment on EC2
>> instances. I know how to run zookeeper on a single EC2 instance manually.
>> But I want to automate such that every time a new instance is spawned,
>> zookeeper will be run on it. There will be around 200 such instances.
>>
>> Any pointers will be helpful.
>>
>> Thanks,
>> Aniket
Re: Running zookeeper on multiple EC2 instances
Posted by Patrick Hunt <ph...@apache.org>.
Take a look at whirr. http://whirr.apache.org/
That said, it's not clear to me what you mean. "every time a new
instance is spawned zk will run on it". Surely you don't mean a zk
server (the max you might run is say 5 or perhaps 7 servers), however
if it's a client then you'd be including the zk jar with your
application code. Perhaps you could elaborate?
Patrick
On Wed, Dec 14, 2011 at 8:44 PM, Aniket Chakrabarti
<ch...@cse.ohio-state.edu> wrote:
> Hi,
>
> I need some suggestion on how to automate zookeeper deployment on EC2
> instances. I know how to run zookeeper on a single EC2 instance manually.
> But I want to automate such that every time a new instance is spawned,
> zookeeper will be run on it. There will be around 200 such instances.
>
> Any pointers will be helpful.
>
> Thanks,
> Aniket