You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Azharuddin (JIRA)" <ji...@apache.org> on 2018/01/18 10:32:00 UTC
[jira] [Updated] (SPARK-23145) How to set Apache Spark Executor
memory
[ https://issues.apache.org/jira/browse/SPARK-23145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Azharuddin updated SPARK-23145:
-------------------------------
Description:
How can I increase the memory available for Apache spark executor nodes?
I have a 2 GB file that is suitable to loading in to [Apache Spark| [https://mindmajix.com|https://mindmajix.com/apache-spark-training]]. I am running apache spark for the moment on 1 machine, so the driver and executor are on the same machine. The machine has 8 GB of memory.
When I try count the lines of the file after setting the file to be cached in memory I get these errors:
{\{2014-10-25 22:25:12 WARN CacheManager:71 - Not enough space to cache partition rdd_1_1 in memory! Free memory is 278099801 bytes. }}
I looked at the documentation [here|http://spark.apache.org/docs/latest/configuration.html] and set {{spark.executor.memory}} to {{4g}} in {{$SPARK_HOME/conf/spark-defaults.conf}}
The UI shows this variable is set in the Spark Environment. You can find screenshot [here|https://drive.google.com/file/d/0B0B_O5bxDDlsc3JmY0xfbjFtN0k/view?usp=sharing]
However when I go to the [Executor tab|https://drive.google.com/file/d/0B0B_O5bxDDlsV25ka2lHd1ZFNzA/view?usp=sharing] the memory limit for my single Executor is still set to 265.4 MB. I also still get the same error.
I tried various things mentioned [here|https://stackoverflow.com/questions/24242060/how-to-change-memory-per-node-for-apache-spark-worker] but I still get the error and don't have a clear idea where I should change the setting.
I am running my code interactively from the spark-shell
was:
How can I increase the memory available for Apache spark executor nodes?
I have a 2 GB file that is suitable to loading in to [Apache Spark|[https://mindmajix.com|https://mindmajix.com/apache-spark-training]]. I am running apache spark for the moment on 1 machine, so the driver and executor are on the same machine. The machine has 8 GB of memory.
When I try count the lines of the file after setting the file to be cached in memory I get these errors:
{{2014-10-25 22:25:12 WARN CacheManager:71 - Not enough space to cache partition rdd_1_1 in memory! Free memory is 278099801 bytes. }}
I looked at the documentation [here|http://spark.apache.org/docs/latest/configuration.html] and set {{spark.executor.memory}} to {{4g}} in {{$SPARK_HOME/conf/spark-defaults.conf}}
The UI shows this variable is set in the Spark Environment. You can find screenshot [here|https://drive.google.com/file/d/0B0B_O5bxDDlsc3JmY0xfbjFtN0k/view?usp=sharing]
However when I go to the [Executor tab|https://drive.google.com/file/d/0B0B_O5bxDDlsV25ka2lHd1ZFNzA/view?usp=sharing] the memory limit for my single Executor is still set to 265.4 MB. I also still get the same error.
I tried various things mentioned [here|https://stackoverflow.com/questions/24242060/how-to-change-memory-per-node-for-apache-spark-worker] but I still get the error and don't have a clear idea where I should change the setting.
I am running my code interactively from the spark-shell
> How to set Apache Spark Executor memory
> ---------------------------------------
>
> Key: SPARK-23145
> URL: https://issues.apache.org/jira/browse/SPARK-23145
> Project: Spark
> Issue Type: Question
> Components: EC2
> Affects Versions: 2.1.0
> Reporter: Azharuddin
> Priority: Major
> Fix For: 2.1.1
>
> Original Estimate: 954h
> Remaining Estimate: 954h
>
> How can I increase the memory available for Apache spark executor nodes?
> I have a 2 GB file that is suitable to loading in to [Apache Spark| [https://mindmajix.com|https://mindmajix.com/apache-spark-training]]. I am running apache spark for the moment on 1 machine, so the driver and executor are on the same machine. The machine has 8 GB of memory.
> When I try count the lines of the file after setting the file to be cached in memory I get these errors:
> {\{2014-10-25 22:25:12 WARN CacheManager:71 - Not enough space to cache partition rdd_1_1 in memory! Free memory is 278099801 bytes. }}
> I looked at the documentation [here|http://spark.apache.org/docs/latest/configuration.html] and set {{spark.executor.memory}} to {{4g}} in {{$SPARK_HOME/conf/spark-defaults.conf}}
> The UI shows this variable is set in the Spark Environment. You can find screenshot [here|https://drive.google.com/file/d/0B0B_O5bxDDlsc3JmY0xfbjFtN0k/view?usp=sharing]
> However when I go to the [Executor tab|https://drive.google.com/file/d/0B0B_O5bxDDlsV25ka2lHd1ZFNzA/view?usp=sharing] the memory limit for my single Executor is still set to 265.4 MB. I also still get the same error.
> I tried various things mentioned [here|https://stackoverflow.com/questions/24242060/how-to-change-memory-per-node-for-apache-spark-worker] but I still get the error and don't have a clear idea where I should change the setting.
> I am running my code interactively from the spark-shell
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org