You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by anshu shukla <an...@gmail.com> on 2015/07/08 19:08:50 UTC
Limiting Workers to use only 1 CORE/2GB RAM
I am running storm on cluster with 8 slaves each with 32 GB AM and 8 CORES
, But i want to limit my slaves to *use only 2 GB RAM and 1 core from
each slaves* .
1 - I have enabled only *1 supervisor slot per machine* ,does that means
every worker will be using *only 1 core *.
2- How can i place the memory restriction .
[This is needed to* benchmark spark with storm* ]
--
Thanks & Regards,
Anshu Shukla
Re: Limiting Workers to use only 1 CORE/2GB RAM
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
Storm worker processes, like many other programs is multithreaded and will attempt to use resources like CPU at its full capacity. Unless you set cpu affinity your self, I don't think storm worker process by default is attached to a single CPU.
Thanks
Parth
From: "Grant Overby (groverby)" <gr...@cisco.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Wednesday, July 8, 2015 at 7:24 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Limiting Workers to use only 1 CORE/2GB RAM
To get the correct PID (replace user name as appropriate): `su - storm -c jps | grep worker | cut -d ' ' -f 1`
Re: Limiting Workers to use only 1 CORE/2GB RAM
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
To clarify , it basically depends largely on your os and jvm what they do with a single jvm process and thread scheduling. So its best to test it out.
Thanks
Parth
From: "Grant Overby (groverby)" <gr...@cisco.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Wednesday, July 8, 2015 at 7:26 PM
To: "Grant Overby (groverby)" <gr...@cisco.com>>, "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Limiting Workers to use only 1 CORE/2GB RAM
>> This is needed to benchmark spark with storm
If possible, please share your test case and results with us.
[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]
Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
groverby@cisco.com<ma...@cisco.com>
Mobile: 865 724 4910
[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.
From: "Grant Overby (groverby)" <gr...@cisco.com>>
Date: Wednesday, July 8, 2015 at 2:24 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Limiting Workers to use only 1 CORE/2GB RAM
To limit the worker process to a single cpu: http://linux.die.net/man/1/taskset
To get the correct PID (replace user name as appropriate): `su - storm -c jps | grep worker | cut -d ' ' -f 1`
[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]
Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
groverby@cisco.com<ma...@cisco.com>
Mobile: 865 724 4910
[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.
From: Parth Brahmbhatt <pb...@hortonworks.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Wednesday, July 8, 2015 at 1:22 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Limiting Workers to use only 1 CORE/2GB RAM
The memory restrictions are easy you can set worker.childopts: "-Xmx1024m" and that will ensure the workers don't go over 1GB. Setting 1 supervisor slot will mean you will only run 1 worker per machine. Note that supervisor process will still use other memory and you can control its memory by specifying supervisor.childopts: "-Xmx1024m".
I don't think there is an easy way to limit the usage of number of cores.
Thanks
Parth
From: anshu shukla <an...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Wednesday, July 8, 2015 at 6:08 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>, "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>
Subject: Limiting Workers to use only 1 CORE/2GB RAM
I am running storm on cluster with 8 slaves each with 32 GB AM and 8 CORES , But i want to limit my slaves to use only 2 GB RAM and 1 core from each slaves .
1 - I have enabled only 1 supervisor slot per machine ,does that means every worker will be using only 1 core .
2- How can i place the memory restriction .
[This is needed to benchmark spark with storm ]
--
Thanks & Regards,
Anshu Shukla
Re: Limiting Workers to use only 1 CORE/2GB RAM
Posted by "Grant Overby (groverby)" <gr...@cisco.com>.
>> This is needed to benchmark spark with storm
If possible, please share your test case and results with us.
[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]
Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
groverby@cisco.com<ma...@cisco.com>
Mobile: 865 724 4910
[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.
From: "Grant Overby (groverby)" <gr...@cisco.com>>
Date: Wednesday, July 8, 2015 at 2:24 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Limiting Workers to use only 1 CORE/2GB RAM
To limit the worker process to a single cpu: http://linux.die.net/man/1/taskset
To get the correct PID (replace user name as appropriate): `su - storm -c jps | grep worker | cut -d ' ' -f 1`
[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]
Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
groverby@cisco.com<ma...@cisco.com>
Mobile: 865 724 4910
[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.
From: Parth Brahmbhatt <pb...@hortonworks.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Wednesday, July 8, 2015 at 1:22 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Limiting Workers to use only 1 CORE/2GB RAM
The memory restrictions are easy you can set worker.childopts: "-Xmx1024m” and that will ensure the workers don’t go over 1GB. Setting 1 supervisor slot will mean you will only run 1 worker per machine. Note that supervisor process will still use other memory and you can control its memory by specifying supervisor.childopts: “-Xmx1024m”.
I don’t think there is an easy way to limit the usage of number of cores.
Thanks
Parth
From: anshu shukla <an...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Wednesday, July 8, 2015 at 6:08 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>, "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>
Subject: Limiting Workers to use only 1 CORE/2GB RAM
I am running storm on cluster with 8 slaves each with 32 GB AM and 8 CORES , But i want to limit my slaves to use only 2 GB RAM and 1 core from each slaves .
1 - I have enabled only 1 supervisor slot per machine ,does that means every worker will be using only 1 core .
2- How can i place the memory restriction .
[This is needed to benchmark spark with storm ]
--
Thanks & Regards,
Anshu Shukla
Re: Limiting Workers to use only 1 CORE/2GB RAM
Posted by "Grant Overby (groverby)" <gr...@cisco.com>.
To limit the worker process to a single cpu: http://linux.die.net/man/1/taskset
To get the correct PID (replace user name as appropriate): `su - storm -c jps | grep worker | cut -d ' ' -f 1`
[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]
Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
groverby@cisco.com<ma...@cisco.com>
Mobile: 865 724 4910
[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.
From: Parth Brahmbhatt <pb...@hortonworks.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Wednesday, July 8, 2015 at 1:22 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Limiting Workers to use only 1 CORE/2GB RAM
The memory restrictions are easy you can set worker.childopts: "-Xmx1024m” and that will ensure the workers don’t go over 1GB. Setting 1 supervisor slot will mean you will only run 1 worker per machine. Note that supervisor process will still use other memory and you can control its memory by specifying supervisor.childopts: “-Xmx1024m”.
I don’t think there is an easy way to limit the usage of number of cores.
Thanks
Parth
From: anshu shukla <an...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Wednesday, July 8, 2015 at 6:08 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>, "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>
Subject: Limiting Workers to use only 1 CORE/2GB RAM
I am running storm on cluster with 8 slaves each with 32 GB AM and 8 CORES , But i want to limit my slaves to use only 2 GB RAM and 1 core from each slaves .
1 - I have enabled only 1 supervisor slot per machine ,does that means every worker will be using only 1 core .
2- How can i place the memory restriction .
[This is needed to benchmark spark with storm ]
--
Thanks & Regards,
Anshu Shukla
Re: Limiting Workers to use only 1 CORE/2GB RAM
Posted by anshu shukla <an...@gmail.com>.
Thanks for help !
Does running only *one worker on every machine* will not make it to
run on *single
core* .(By assigning only one supervisor slot to each machine ) ??
On Wed, Jul 8, 2015 at 10:52 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:
> The memory restrictions are easy you can set worker.childopts:
> "-Xmx1024m” and that will ensure the workers don’t go over 1GB. Setting 1
> supervisor slot will mean you will only run 1 worker per machine. Note that
> supervisor process will still use other memory and you can control its
> memory by specifying supervisor.childopts: “-Xmx1024m”.
>
> I don’t think there is an easy way to limit the usage of number of cores.
>
> Thanks
> Parth
>
> From: anshu shukla <an...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Wednesday, July 8, 2015 at 6:08 PM
> To: "user@storm.apache.org" <us...@storm.apache.org>, "dev@storm.apache.org"
> <de...@storm.apache.org>
> Subject: Limiting Workers to use only 1 CORE/2GB RAM
>
> I am running storm on cluster with 8 slaves each with 32 GB AM and 8
> CORES , But i want to limit my slaves to *use only 2 GB RAM and 1 core
> from each slaves* .
>
> 1 - I have enabled only *1 supervisor slot per machine* ,does that
> means every worker will be using *only 1 core *.
>
> 2- How can i place the memory restriction .
>
> [This is needed to* benchmark spark with storm* ]
>
> --
> Thanks & Regards,
> Anshu Shukla
>
--
Thanks & Regards,
Anshu Shukla
Re: Limiting Workers to use only 1 CORE/2GB RAM
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
The memory restrictions are easy you can set worker.childopts: "-Xmx1024m" and that will ensure the workers don't go over 1GB. Setting 1 supervisor slot will mean you will only run 1 worker per machine. Note that supervisor process will still use other memory and you can control its memory by specifying supervisor.childopts: "-Xmx1024m".
I don't think there is an easy way to limit the usage of number of cores.
Thanks
Parth
From: anshu shukla <an...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Wednesday, July 8, 2015 at 6:08 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>, "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>
Subject: Limiting Workers to use only 1 CORE/2GB RAM
I am running storm on cluster with 8 slaves each with 32 GB AM and 8 CORES , But i want to limit my slaves to use only 2 GB RAM and 1 core from each slaves .
1 - I have enabled only 1 supervisor slot per machine ,does that means every worker will be using only 1 core .
2- How can i place the memory restriction .
[This is needed to benchmark spark with storm ]
--
Thanks & Regards,
Anshu Shukla