Cannot change the memory of workers

[Jia Yu <ji...@asu.edu>]

Hi guys,

Currently I am running Spark program on Amazon EC2. Each worker has around
(less than but near to )2 gb memory.

By default, I can see each worker is allocated 976 mb memory as the table
shows below on Spark WEB UI. I know this value is from (Total memory minus
1 GB). But I want more than 1 gb in each of my worker.

AddressStateCoresMemory

ALIVE1 (0 Used)976.0 MB (0.0 B Used)Based on the instruction on Spark
website, I made "export SPARK_WORKER_MEMORY=1g" in spark-env.sh. But it
doesn't work. BTW, I can set "SPARK_EXECUTOR_MEMORY=1g" and it works.

Can anyone help me? Is there a requirement that one worker must maintain 1
gb memory for itself aside from the memory for Spark?

Thanks,
Jia

Re: Cannot change the memory of workers

[Akhil Das <ak...@sigmoidanalytics.com>]

If you want to use 2g of memory on each worker, you can simply export
SPARK_WORKER_MEMORY=2g inside your spark-env.sh on all machine in the
cluster.

Thanks
Best Regards

On Wed, Apr 8, 2015 at 7:27 AM, Jia Yu <ji...@asu.edu> wrote:

> Hi guys,
>
> Currently I am running Spark program on Amazon EC2. Each worker has around
> (less than but near to )2 gb memory.
>
> By default, I can see each worker is allocated 976 mb memory as the table
> shows below on Spark WEB UI. I know this value is from (Total memory minus
> 1 GB). But I want more than 1 gb in each of my worker.
>
> AddressStateCoresMemory
>
> ALIVE1 (0 Used)976.0 MB (0.0 B Used)Based on the instruction on Spark
> website, I made "export SPARK_WORKER_MEMORY=1g" in spark-env.sh. But it
> doesn't work. BTW, I can set "SPARK_EXECUTOR_MEMORY=1g" and it works.
>
> Can anyone help me? Is there a requirement that one worker must maintain 1
> gb memory for itself aside from the memory for Spark?
>
> Thanks,
> Jia
>