You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ian Ferreira <ia...@hotmail.com> on 2014/05/07 02:29:18 UTC
Easy one
Hi there,
Why can¹t I seem to kick the executor memory higher? See below from EC2
deployment using m1.large
And in the spark-env.sh
export SPARK_MEM=6154m
And in the spark context
sconf.setExecutorEnv("spark.executor.memory", "4g²)
Cheers
- Ian
Re: Easy one
Posted by Ian Ferreira <ia...@hotmail.com>.
Spoke to soon, while I can get the worker memory up, I can¹t seem to
increase the executor memory, still seems locked in a 512MB
Even with context set up like so
sconf.setExecutorEnv("spark.executor.memory", "1g²)
From: Ian Ferreira <ia...@hotmail.com>
Date: Tuesday, May 6, 2014 at 6:10 PM
To: <us...@spark.apache.org>
Subject: Re: Easy one
Thanks!
From: Aaron Davidson <il...@gmail.com>
Reply-To: <us...@spark.apache.org>
Date: Tuesday, May 6, 2014 at 5:32 PM
To: <us...@spark.apache.org>
Subject: Re: Easy one
If you're using standalone mode, you need to make sure the Spark Workers
know about the extra memory. This can be configured in spark-env.sh on the
workers as
export SPARK_WORKER_MEMORY=4g
On Tue, May 6, 2014 at 5:29 PM, Ian Ferreira <ia...@hotmail.com>
wrote:
> Hi there,
>
> Why can¹t I seem to kick the executor memory higher? See below from EC2
> deployment using m1.large
>
>
> And in the spark-env.sh
> export SPARK_MEM=6154m
>
>
> And in the spark context
> sconf.setExecutorEnv("spark.executor.memory", "4g²)
>
> Cheers
> - Ian
>
Re: Easy one
Posted by Ian Ferreira <ia...@hotmail.com>.
Thanks!
From: Aaron Davidson <il...@gmail.com>
Reply-To: <us...@spark.apache.org>
Date: Tuesday, May 6, 2014 at 5:32 PM
To: <us...@spark.apache.org>
Subject: Re: Easy one
If you're using standalone mode, you need to make sure the Spark Workers
know about the extra memory. This can be configured in spark-env.sh on the
workers as
export SPARK_WORKER_MEMORY=4g
On Tue, May 6, 2014 at 5:29 PM, Ian Ferreira <ia...@hotmail.com>
wrote:
> Hi there,
>
> Why can¹t I seem to kick the executor memory higher? See below from EC2
> deployment using m1.large
>
>
> And in the spark-env.sh
> export SPARK_MEM=6154m
>
>
> And in the spark context
> sconf.setExecutorEnv("spark.executor.memory", "4g²)
>
> Cheers
> - Ian
>
Re: Easy one
Posted by Aaron Davidson <il...@gmail.com>.
If you're using standalone mode, you need to make sure the Spark Workers
know about the extra memory. This can be configured in spark-env.sh on the
workers as
export SPARK_WORKER_MEMORY=4g
On Tue, May 6, 2014 at 5:29 PM, Ian Ferreira <ia...@hotmail.com>wrote:
> Hi there,
>
> Why can’t I seem to kick the executor memory higher? See below from EC2
> deployment using m1.large
>
>
> And in the spark-env.sh
> export SPARK_MEM=6154m
>
>
> And in the spark context
> sconf.setExecutorEnv("spark.executor.memory", "4g”)
>
> Cheers
> - Ian
>
>
Re: Easy one
Posted by Laeeq Ahmed <la...@yahoo.com>.
Hi Ian,
Don't use SPARK_MEM in spark-env.sh. It will get it set for all of your jobs. The better way is to use only the second option sconf.setExecutorEnv("spark.executor.memory", "4g”) i.e. set it in the driver program. In this way every job will have memory according to requirment.
For example you have 4gb memory on each worker node, you won't be able to run more than one job with 4gb given to each. Two jobs with 2GB(or slightly less) each will work.
Laeeq
On Wednesday, May 7, 2014 2:29 AM, Ian Ferreira <ia...@hotmail.com> wrote:
Hi there,
Why can’t I seem to kick the executor memory higher? See below from EC2 deployment using m1.large
And in the spark-env.sh
export SPARK_MEM=6154m
And in the spark context
sconf.setExecutorEnv("spark.executor.memory", "4g”)
Cheers
- Ian