Set memory when using local[k]

[agg <ag...@gmail.com>]

Hi guys,

I would like to run some algorithms on a single machine, using the local[k]
option.  I cannot however seem to set the amount of memory available, it
always defaults to 700mb.  I've tried setting spark.executor.memory before
creating the spark context.

Is there any other way to increase the amount of memory when using the
local[k] option?

Thanks!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Set-memory-when-using-local-k-tp1945.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Set memory when using local[k]

[Aaron Davidson <il...@gmail.com>]

local[k] simply creates k threads in the same process, so you just need to
set the memory of the spawning process, using standard Java options.

Also, avoid SPARK_MEM when possible, as it will be deprecated by 1.0.0. (In
0.9.0 and prior versions, you still need SPARK_MEM to set shell memory,
though.)


On Sun, Feb 23, 2014 at 1:19 PM, agg <ag...@gmail.com> wrote:

> I am trying to run a stand-alone job, using sbt and not the spark
> shell...any
> thoughts for this?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Set-memory-when-using-local-k-tp1945p1950.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>

Re: Set memory when using local[k]

[agg <ag...@gmail.com>]

I am trying to run a stand-alone job, using sbt and not the spark shell...any
thoughts for this?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Set-memory-when-using-local-k-tp1945p1950.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Set memory when using local[k]

[Gerard Maas <ge...@gmail.com>]

You can use the environment variable SPARK_MEM to set the amount of RAM
that the spark shell will use:

eg.: Use 8GB of memory:

SPARK_MEM=8G ./bin/spark-shell

-kr, Gerard.




On Sun, Feb 23, 2014 at 9:50 PM, agg <ag...@gmail.com> wrote:

> Hi guys,
>
> I would like to run some algorithms on a single machine, using the local[k]
> option.  I cannot however seem to set the amount of memory available, it
> always defaults to 700mb.  I've tried setting spark.executor.memory before
> creating the spark context.
>
> Is there any other way to increase the amount of memory when using the
> local[k] option?
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Set-memory-when-using-local-k-tp1945.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>