configuring max sum of cores and memory in cluster through command line

[Alexander Waldin <aw...@inflection.com>]

Hi,

I'd like to specify the total sum of cores / memory as command line 
arguments with spark-submit. That is, I'd like to set 
yarn.nodemanager.resource.memory-mb and the 
yarn.nodemanager.resource.cpu-vcores parameters as described in this 
blog 
<http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/> 
post.

when submitting through the command line, what is the correct way to do 
it? Is it:

--conf spark.yarn.nodemanager.resource.memory-mb=54g
--conf spark.yarn.nodemanager.resource.cpu-vcores=31

or

--conf yarn.nodemanager.resource.memory-mb=54g
--conf yarn.nodemanager.resource.cpu-vcores=31


or something else? I tried these, and I tried looking in the 
ResourceManager UI to see if they were set, but couldn't find them.

Thanks!

Alexander

Re: configuring max sum of cores and memory in cluster through command line

[Ruslan Dautkhanov <da...@gmail.com>]

It's not possible to specify YARN RM paramers at command line of
spark-submit time. You have to specify all resources that are available on
your cluster to YARN upfront. If you want to limit amount of resource
available for your Spark job, consider using YARN dynamic resource pools
instead

http://www.cloudera.com/content/cloudera/en/documentation/cloudera-manager/v5-1-x/Cloudera-Manager-Managing-Clusters/cm5mc_resource_pools.html




-- 
Ruslan Dautkhanov

On Thu, Jul 2, 2015 at 4:20 PM, Alexander Waldin <aw...@inflection.com>
wrote:

>  Hi,
>
> I'd like to specify the total sum of cores / memory as command line
> arguments with spark-submit. That is, I'd like to set
> yarn.nodemanager.resource.memory-mb and the
> yarn.nodemanager.resource.cpu-vcores parameters as described in this blog
> <http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/>
> post.
>
> when submitting through the command line, what is the correct way to do
> it? Is it:
>
> --conf spark.yarn.nodemanager.resource.memory-mb=54g
> --conf spark.yarn.nodemanager.resource.cpu-vcores=31
>
> or
>
> --conf yarn.nodemanager.resource.memory-mb=54g
> --conf yarn.nodemanager.resource.cpu-vcores=31
>
>
> or something else? I tried these, and I tried looking in the
> ResourceManager UI to see if they were set, but couldn't find them.
>
> Thanks!
>
> Alexander
>