You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by centerqi hu <ce...@gmail.com> on 2014/08/21 03:56:56 UTC

Spark memory settings on yarn

Spark memory settings let me very misunderstanding.

My code is as follows.

spark-1.0.2-bin-2.4.1/bin/spark-submit --class SimpleApp \
--master yarn \
--deploy-mode cluster \
--queue sls_queue_1 \
--num-executors 3 \
--driver-memory 6g \
--executor-memory 10g \
--executor-cores 5 \
target/scala-2.10/simple-project_2.10-1.0.jar \
/user/www/abc/output/2014-08-*/*


Set executor-memory 10g,
But, see the nodemanager java process.
Why is Xmx3072m instead of 10G?

jdk1.7.0_67//bin/java -Djava.net.preferIPv4Stack=true
-Dhadoop.metrics.log.level=WARN -Xmx3072m
-Djava.io.tmpdir=/data/hadoop/nodemanager/usercache/www/appcache/application_1408182086233_0013/container_1408182086233_0013_01_000004/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/data/hadoop/logs/nodemanager/application_1408182086233_0013/container_1408182086233_0013_01_000004
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
org.apache.hadoop.mapred.YarnChild 10.1.13.11 35389
attempt_1408182086233_0013_m_000002_0 4


Thx

centerqi@gmail.com|齐忠

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Spark memory settings on yarn

Posted by centerqi hu <ce...@gmail.com>.
thanks Marcelo Vanzin
I see.
Org.apache.spark.executor.CoarseGrainedExecutorBackend will be executed

Thanks

2014-08-21 12:34 GMT+08:00 Marcelo Vanzin <va...@cloudera.com>:
> That command line you mention in your e-mail doesn't look like
> something started by Spark. Spark would start one of
> ApplicationMaster, ExecutableRunner or CoarseGrainedSchedulerBackend,
> not "org.apache.hadoop.mapred.YarnChild".
>
> On Wed, Aug 20, 2014 at 6:56 PM, centerqi hu <ce...@gmail.com> wrote:
>> Spark memory settings let me very misunderstanding.
>>
>> My code is as follows.
>>
>> spark-1.0.2-bin-2.4.1/bin/spark-submit --class SimpleApp \
>> --master yarn \
>> --deploy-mode cluster \
>> --queue sls_queue_1 \
>> --num-executors 3 \
>> --driver-memory 6g \
>> --executor-memory 10g \
>> --executor-cores 5 \
>> target/scala-2.10/simple-project_2.10-1.0.jar \
>> /user/www/abc/output/2014-08-*/*
>>
>>
>> Set executor-memory 10g,
>> But, see the nodemanager java process.
>> Why is Xmx3072m instead of 10G?
>>
>> jdk1.7.0_67//bin/java -Djava.net.preferIPv4Stack=true
>> -Dhadoop.metrics.log.level=WARN -Xmx3072m
>> -Djava.io.tmpdir=/data/hadoop/nodemanager/usercache/www/appcache/application_1408182086233_0013/container_1408182086233_0013_01_000004/tmp
>> -Dlog4j.configuration=container-log4j.properties
>> -Dyarn.app.container.log.dir=/data/hadoop/logs/nodemanager/application_1408182086233_0013/container_1408182086233_0013_01_000004
>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>> org.apache.hadoop.mapred.YarnChild 10.1.13.11 35389
>> attempt_1408182086233_0013_m_000002_0 4
>>
>>
>> Thx
>>
>> centerqi@gmail.com|齐忠
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
>
>
> --
> Marcelo



-- 
centerqi@gmail.com|齐忠

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Spark memory settings on yarn

Posted by Marcelo Vanzin <va...@cloudera.com>.
That command line you mention in your e-mail doesn't look like
something started by Spark. Spark would start one of
ApplicationMaster, ExecutableRunner or CoarseGrainedSchedulerBackend,
not "org.apache.hadoop.mapred.YarnChild".

On Wed, Aug 20, 2014 at 6:56 PM, centerqi hu <ce...@gmail.com> wrote:
> Spark memory settings let me very misunderstanding.
>
> My code is as follows.
>
> spark-1.0.2-bin-2.4.1/bin/spark-submit --class SimpleApp \
> --master yarn \
> --deploy-mode cluster \
> --queue sls_queue_1 \
> --num-executors 3 \
> --driver-memory 6g \
> --executor-memory 10g \
> --executor-cores 5 \
> target/scala-2.10/simple-project_2.10-1.0.jar \
> /user/www/abc/output/2014-08-*/*
>
>
> Set executor-memory 10g,
> But, see the nodemanager java process.
> Why is Xmx3072m instead of 10G?
>
> jdk1.7.0_67//bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx3072m
> -Djava.io.tmpdir=/data/hadoop/nodemanager/usercache/www/appcache/application_1408182086233_0013/container_1408182086233_0013_01_000004/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/data/hadoop/logs/nodemanager/application_1408182086233_0013/container_1408182086233_0013_01_000004
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 10.1.13.11 35389
> attempt_1408182086233_0013_m_000002_0 4
>
>
> Thx
>
> centerqi@gmail.com|齐忠
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>



-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org