You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Sai Dilip Reddy Kiralam <dk...@aadhya-analytics.com> on 2016/04/11 15:10:14 UTC
How to set number of slots in storm.yaml
Hi All,
I'm using a 16 core cpu as my supervisor node.My doubt is How many
supervisor.slots should I set in supervisor storm.yaml conf file ?Is there
any formula to set no of slots based on the cpu core.
Thanks in Advance
*Best regards,*
*K.Sai Dilip Reddy.*
Re: How to set number of slots in storm.yaml
Posted by Andrey Dudin <do...@gmail.com>.
Hi,
It's like virtualization. If you have many small topologies that need few
CPU resources, you can add more processes per CPU. It you have big
topologies that need a lot of CPU resources, you need to decrease number of
process per cpu. I advise to start from 1.2-1.5 process per CPU and compare
with 1 process per cpu and 2 process per cpu.
2016-04-12 7:44 GMT+03:00 Sai Dilip Reddy Kiralam <
dkiralam@aadhya-analytics.com>:
> Hi guys,
>
> Thank you for your suggestions but I'm not clear with the
> explanation.andrey please let me know one thing '*what do you mean by
> "depend by code"* ' ? Depend on code in the sense does it depends on
> number of bolts and spouts I'm using ?if yes can you give me a direct
> answer using an example having number of bolts and spouts.How many
> supervisor.slots should I consider in 16 core cpu?Answer to my question
> Still remains as a lock.
> machine.
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>
> On Mon, Apr 11, 2016 at 9:29 PM, Andrey Dudin <do...@gmail.com> wrote:
>
>> Hi Nikos.
>>
>> Yes, you are right, but sometimes java application show better
>> performance under high load when used 1.2-1.5 process per core. Thicker
>> loading is not always bad.
>>
>> 2016-04-11 18:23 GMT+03:00 Nikos R. Katsipoulakis <ni...@gmail.com>
>> :
>>
>>> Hello both,
>>>
>>> Andrey, I believe its better to use one executor thread per core and
>>> only one worker process per socket. This way, the executor threads
>>> communicate with each other using LMAX Disruptor queues (therefore less
>>> latency) and Netty is used only when a message needs to be sent to another
>>> executor thread on a different machine.
>>>
>>> Also, if you need a lot of memory, you can start your worker process
>>> with as much Heap memory you want.
>>>
>>> Cheers,
>>> Nikos
>>>
>>> On Mon, Apr 11, 2016 at 9:15 AM, Andrey Dudin <do...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> It's depend by code. Usually starting from 1-1,5 process per core.
>>>>
>>>> 2016-04-11 16:10 GMT+03:00 Sai Dilip Reddy Kiralam <
>>>> dkiralam@aadhya-analytics.com>:
>>>>
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I'm using a 16 core cpu as my supervisor node.My doubt is How many
>>>>> supervisor.slots should I set in supervisor storm.yaml conf file ?Is there
>>>>> any formula to set no of slots based on the cpu core.
>>>>>
>>>>> Thanks in Advance
>>>>>
>>>>>
>>>>> *Best regards,*
>>>>>
>>>>> *K.Sai Dilip Reddy.*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> С уважением Дудин Андрей
>>>>
>>>
>>>
>>>
>>> --
>>> Nikos R. Katsipoulakis,
>>> Department of Computer Science
>>> University of Pittsburgh
>>>
>>
>>
>>
>> --
>> С уважением Дудин Андрей
>>
>
>
--
С уважением Дудин Андрей
Re: How to set number of slots in storm.yaml
Posted by Nathan Leung <nc...@gmail.com>.
I would stick with 4 and change it only if you have a good reason to do
so. Too many worker processes can be detrimental. Each storm process is
by nature highly threaded. There's not much benefit to making so many
processes, unless you need really fine grained fault tolerance.
See also:
https://mail-archives.apache.org/mod_mbox/storm-user/201603.mbox/browser
There are other discussions on the mailing list as well but I'd have to dig
a bit more to find them.
On Tue, Apr 12, 2016 at 4:56 AM, Andrey Dudin <do...@gmail.com> wrote:
> Try to use 1.2-1.5 process per cpu and watch. Count of process per
> CPU strongly depends on the application. You can find best coefficient only
> empirically.
>
> 2016-04-12 9:43 GMT+03:00 sujitha chinnu <ch...@gmail.com>:
>
>> Hai all,
>>
>> I too have the same question.In storm.yaml file we have to
>> set the supervisor slots. At present I am setting 4 slots for 4-core cpu as:
>>
>> *supervisor.slots.ports:*
>> - 6700
>> - 6701
>> - 6702
>> - 6703
>>
>> If i am using 16-core cpu then how many supervisor slots should i use?
>>
>> Regards,
>> sujitha.
>>
>
>
>
> --
> С уважением Дудин Андрей
>
Re: How to set number of slots in storm.yaml
Posted by Andrey Dudin <do...@gmail.com>.
Try to use 1.2-1.5 process per cpu and watch. Count of process per
CPU strongly depends on the application. You can find best coefficient only
empirically.
2016-04-12 9:43 GMT+03:00 sujitha chinnu <ch...@gmail.com>:
> Hai all,
>
> I too have the same question.In storm.yaml file we have to
> set the supervisor slots. At present I am setting 4 slots for 4-core cpu as:
>
> *supervisor.slots.ports:*
> - 6700
> - 6701
> - 6702
> - 6703
>
> If i am using 16-core cpu then how many supervisor slots should i use?
>
> Regards,
> sujitha.
>
--
С уважением Дудин Андрей
Re: How to set number of slots in storm.yaml
Posted by sujitha chinnu <ch...@gmail.com>.
Hai all,
I too have the same question.In storm.yaml file we have to set
the supervisor slots. At present I am setting 4 slots for 4-core cpu as:
*supervisor.slots.ports:*
- 6700
- 6701
- 6702
- 6703
If i am using 16-core cpu then how many supervisor slots should i use?
Regards,
sujitha.
Re: How to set number of slots in storm.yaml
Posted by Sai Dilip Reddy Kiralam <dk...@aadhya-analytics.com>.
Hi guys,
Thank you for your suggestions but I'm not clear with the
explanation.andrey please let me know one thing '*what do you mean by
"depend by code"* ' ? Depend on code in the sense does it depends on number
of bolts and spouts I'm using ?if yes can you give me a direct answer using
an example having number of bolts and spouts.How many supervisor.slots
should I consider in 16 core cpu?Answer to my question Still remains as a
lock.
machine.
*Best regards,*
*K.Sai Dilip Reddy.*
On Mon, Apr 11, 2016 at 9:29 PM, Andrey Dudin <do...@gmail.com> wrote:
> Hi Nikos.
>
> Yes, you are right, but sometimes java application show better performance
> under high load when used 1.2-1.5 process per core. Thicker loading is not
> always bad.
>
> 2016-04-11 18:23 GMT+03:00 Nikos R. Katsipoulakis <ni...@gmail.com>:
>
>> Hello both,
>>
>> Andrey, I believe its better to use one executor thread per core and only
>> one worker process per socket. This way, the executor threads communicate
>> with each other using LMAX Disruptor queues (therefore less latency) and
>> Netty is used only when a message needs to be sent to another executor
>> thread on a different machine.
>>
>> Also, if you need a lot of memory, you can start your worker process with
>> as much Heap memory you want.
>>
>> Cheers,
>> Nikos
>>
>> On Mon, Apr 11, 2016 at 9:15 AM, Andrey Dudin <do...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> It's depend by code. Usually starting from 1-1,5 process per core.
>>>
>>> 2016-04-11 16:10 GMT+03:00 Sai Dilip Reddy Kiralam <
>>> dkiralam@aadhya-analytics.com>:
>>>
>>>>
>>>> Hi All,
>>>>
>>>> I'm using a 16 core cpu as my supervisor node.My doubt is How many
>>>> supervisor.slots should I set in supervisor storm.yaml conf file ?Is there
>>>> any formula to set no of slots based on the cpu core.
>>>>
>>>> Thanks in Advance
>>>>
>>>>
>>>> *Best regards,*
>>>>
>>>> *K.Sai Dilip Reddy.*
>>>>
>>>
>>>
>>>
>>> --
>>> С уважением Дудин Андрей
>>>
>>
>>
>>
>> --
>> Nikos R. Katsipoulakis,
>> Department of Computer Science
>> University of Pittsburgh
>>
>
>
>
> --
> С уважением Дудин Андрей
>
Re: How to set number of slots in storm.yaml
Posted by Andrey Dudin <do...@gmail.com>.
Hi Nikos.
Yes, you are right, but sometimes java application show better performance
under high load when used 1.2-1.5 process per core. Thicker loading is not
always bad.
2016-04-11 18:23 GMT+03:00 Nikos R. Katsipoulakis <ni...@gmail.com>:
> Hello both,
>
> Andrey, I believe its better to use one executor thread per core and only
> one worker process per socket. This way, the executor threads communicate
> with each other using LMAX Disruptor queues (therefore less latency) and
> Netty is used only when a message needs to be sent to another executor
> thread on a different machine.
>
> Also, if you need a lot of memory, you can start your worker process with
> as much Heap memory you want.
>
> Cheers,
> Nikos
>
> On Mon, Apr 11, 2016 at 9:15 AM, Andrey Dudin <do...@gmail.com> wrote:
>
>> Hi,
>> It's depend by code. Usually starting from 1-1,5 process per core.
>>
>> 2016-04-11 16:10 GMT+03:00 Sai Dilip Reddy Kiralam <
>> dkiralam@aadhya-analytics.com>:
>>
>>>
>>> Hi All,
>>>
>>> I'm using a 16 core cpu as my supervisor node.My doubt is How many
>>> supervisor.slots should I set in supervisor storm.yaml conf file ?Is there
>>> any formula to set no of slots based on the cpu core.
>>>
>>> Thanks in Advance
>>>
>>>
>>> *Best regards,*
>>>
>>> *K.Sai Dilip Reddy.*
>>>
>>
>>
>>
>> --
>> С уважением Дудин Андрей
>>
>
>
>
> --
> Nikos R. Katsipoulakis,
> Department of Computer Science
> University of Pittsburgh
>
--
С уважением Дудин Андрей
Re: How to set number of slots in storm.yaml
Posted by "Nikos R. Katsipoulakis" <ni...@gmail.com>.
Hello both,
Andrey, I believe its better to use one executor thread per core and only
one worker process per socket. This way, the executor threads communicate
with each other using LMAX Disruptor queues (therefore less latency) and
Netty is used only when a message needs to be sent to another executor
thread on a different machine.
Also, if you need a lot of memory, you can start your worker process with
as much Heap memory you want.
Cheers,
Nikos
On Mon, Apr 11, 2016 at 9:15 AM, Andrey Dudin <do...@gmail.com> wrote:
> Hi,
> It's depend by code. Usually starting from 1-1,5 process per core.
>
> 2016-04-11 16:10 GMT+03:00 Sai Dilip Reddy Kiralam <
> dkiralam@aadhya-analytics.com>:
>
>>
>> Hi All,
>>
>> I'm using a 16 core cpu as my supervisor node.My doubt is How many
>> supervisor.slots should I set in supervisor storm.yaml conf file ?Is there
>> any formula to set no of slots based on the cpu core.
>>
>> Thanks in Advance
>>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>
>
>
> --
> С уважением Дудин Андрей
>
--
Nikos R. Katsipoulakis,
Department of Computer Science
University of Pittsburgh
Re: How to set number of slots in storm.yaml
Posted by Andrey Dudin <do...@gmail.com>.
Hi,
It's depend by code. Usually starting from 1-1,5 process per core.
2016-04-11 16:10 GMT+03:00 Sai Dilip Reddy Kiralam <
dkiralam@aadhya-analytics.com>:
>
> Hi All,
>
> I'm using a 16 core cpu as my supervisor node.My doubt is How many
> supervisor.slots should I set in supervisor storm.yaml conf file ?Is there
> any formula to set no of slots based on the cpu core.
>
> Thanks in Advance
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>
--
С уважением Дудин Андрей