[jira] [Created] (SPARK-23145) How to set Apache Spark Executor memory

["Azharuddin (JIRA)" <ji...@apache.org>]

Azharuddin created SPARK-23145:
----------------------------------

             Summary: How to set Apache Spark Executor memory
                 Key: SPARK-23145
                 URL: https://issues.apache.org/jira/browse/SPARK-23145
             Project: Spark
          Issue Type: Question
          Components: EC2
    Affects Versions: 2.1.0
            Reporter: Azharuddin
             Fix For: 2.1.1


How can I increase the memory available for Apache spark executor nodes?

I have a 2 GB file that is suitable to loading in to [Apache Spark|[https://mindmajix.com|https://mindmajix.com/apache-spark-training]]. I am running apache spark for the moment on 1 machine, so the driver and executor are on the same machine. The machine has 8 GB of memory.

When I try count the lines of the file after setting the file to be cached in memory I get these errors:

{{2014-10-25 22:25:12 WARN CacheManager:71 - Not enough space to cache partition rdd_1_1 in memory! Free memory is 278099801 bytes. }}

I looked at the documentation [here|http://spark.apache.org/docs/latest/configuration.html] and set {{spark.executor.memory}} to {{4g}} in {{$SPARK_HOME/conf/spark-defaults.conf}}

The UI shows this variable is set in the Spark Environment. You can find screenshot [here|https://drive.google.com/file/d/0B0B_O5bxDDlsc3JmY0xfbjFtN0k/view?usp=sharing]

However when I go to the [Executor tab|https://drive.google.com/file/d/0B0B_O5bxDDlsV25ka2lHd1ZFNzA/view?usp=sharing] the memory limit for my single Executor is still set to 265.4 MB. I also still get the same error.

I tried various things mentioned [here|https://stackoverflow.com/questions/24242060/how-to-change-memory-per-node-for-apache-spark-worker] but I still get the error and don't have a clear idea where I should change the setting.

I am running my code interactively from the spark-shell



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org