You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by nancy henry <na...@gmail.com> on 2017/02/13 10:27:45 UTC
Lost executor 4 Container killed by YARN for exceeding memory limits.
Hi All,,
I am getting below error while I am trying to join 3 tables which are in
ORC format in hive from 5 10gb tables through hive context in spark
Container killed by YARN for exceeding memory limits. 11.1 GB of 11 GB
physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
17/02/13 02:21:19 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:
Container killed by YARN for exceeding memory limits. 11.1 GB of 11 GB
physical memory used
I am using below memory parameters to launch shell .. what else i could
increase from these parameters or do I need to change any configuration
settings please let me know
spark-shell --master yarn --deploy-mode client --driver-memory 16G
--num-executors 500 executor-cores 7 --executor-memory 10G
Re: Lost executor 4 Container killed by YARN for exceeding memory
limits.
Posted by "Thakrar, Jayesh" <jt...@conversantmedia.com>.
Nancy,
As the message from Spark indicates, spark.shuffle.memoryFraction is no longer used.
It’s a unified heap space for both data caching and other things.
Also, the previous 11 GB was not sufficient, and you are making the executor memory even smaller, not sure how it will work.
From: nancy henry <na...@gmail.com>
Date: Tuesday, February 14, 2017 at 1:04 AM
To: Conversant <jt...@conversantmedia.com>
Cc: Jon Gregg <co...@gmail.com>, "user @spark" <us...@spark.apache.org>
Subject: Re: Lost executor 4 Container killed by YARN for exceeding memory limits.
Hi,
How to set this parameters while launching spark shell
spark.shuffle.memoryFraction=0.5
and
spark.yarn.executor.memoryOverhead=1024
I tried giving like this but I am giving below error
spark-shell --master yarn --deploy-mode client --driver-memory 16G --num-executors 500 executor-cores 4 --executor-memory 7G --conf spark.shuffle.memoryFraction=0.5 --conf spark.yarn.executor.memoryOverhead=1024
Warning
17/02/13 22:42:02 WARN SparkConf: Detected deprecated memory fraction settings: [spark.shuffle.memoryFraction]. As of Spark 1.6, execution and storage memory management are unified. All memory fractions used in the old model are now deprecated and no longer read. If you wish to use the old memory management, you may explicitly enable `spark.memory.useLegacyMode` (not recommended).
On Mon, Feb 13, 2017 at 11:23 PM, Thakrar, Jayesh <jt...@conversantmedia.com>> wrote:
Nancy,
As your log output indicated, your executor 11 GB memory limit.
While you might want to address the root cause/data volume as suggested by Jon, you can do an immediate test by changing your command as follows
spark-shell --master yarn --deploy-mode client --driver-memory 16G --num-executors 500 executor-cores 7 --executor-memory 14G
This essentially increases your executor memory from 11 GB to 14 GB.
Note that it will result in a potentially large footprint - from 500x11 to 500x14 GB.
You may want to consult with your DevOps/Operations/Spark Admin team first.
From: Jon Gregg <co...@gmail.com>>
Date: Monday, February 13, 2017 at 8:58 AM
To: nancy henry <na...@gmail.com>>
Cc: "user @spark" <us...@spark.apache.org>>
Subject: Re: Lost executor 4 Container killed by YARN for exceeding memory limits.
Setting Spark's memoryOverhead configuration variable is recommended in your logs, and has helped me with these issues in the past. Search for "memoryOverhead" here: http://spark.apache.org/docs/latest/running-on-yarn.html
That said, you're running on a huge cluster as it is. If it's possible to filter your tables down before the join (keeping just the rows/columns you need), that may be a better solution.
Jon
On Mon, Feb 13, 2017 at 5:27 AM, nancy henry <na...@gmail.com>> wrote:
Hi All,,
I am getting below error while I am trying to join 3 tables which are in ORC format in hive from 5 10gb tables through hive context in spark
Container killed by YARN for exceeding memory limits. 11.1 GB of 11 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
17/02/13 02:21:19 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 11.1 GB of 11 GB physical memory used
I am using below memory parameters to launch shell .. what else i could increase from these parameters or do I need to change any configuration settings please let me know
spark-shell --master yarn --deploy-mode client --driver-memory 16G --num-executors 500 executor-cores 7 --executor-memory 10G
Re: Lost executor 4 Container killed by YARN for exceeding memory limits.
Posted by nancy henry <na...@gmail.com>.
Hi,
How to set this parameters while launching spark shell
spark.shuffle.memoryFraction=0.5
and
spark.yarn.executor.memoryOverhead=1024
I tried giving like this but I am giving below error
spark-shell --master yarn --deploy-mode client --driver-memory 16G
--num-executors 500 executor-cores 4 --executor-memory 7G --conf
spark.shuffle.memoryFraction=0.5 --conf
spark.yarn.executor.memoryOverhead=1024
Warning
17/02/13 22:42:02 WARN SparkConf: Detected deprecated memory fraction
settings: [spark.shuffle.memoryFraction]. As of Spark 1.6, execution and
storage memory management are unified. All memory fractions used in the old
model are now deprecated and no longer read. If you wish to use the old
memory management, you may explicitly enable `spark.memory.useLegacyMode`
(not recommended).
On Mon, Feb 13, 2017 at 11:23 PM, Thakrar, Jayesh <
jthakrar@conversantmedia.com> wrote:
> Nancy,
>
>
>
> As your log output indicated, your executor 11 GB memory limit.
>
> While you might want to address the root cause/data volume as suggested by
> Jon, you can do an immediate test by changing your command as follows
>
>
>
> spark-shell --master yarn --deploy-mode client --driver-memory 16G
> --num-executors 500 executor-cores 7 --executor-memory 14G
>
>
>
> This essentially increases your executor memory from 11 GB to 14 GB.
>
> Note that it will result in a potentially large footprint - from 500x11 to
> 500x14 GB.
>
> You may want to consult with your DevOps/Operations/Spark Admin team first.
>
>
>
> *From: *Jon Gregg <co...@gmail.com>
> *Date: *Monday, February 13, 2017 at 8:58 AM
> *To: *nancy henry <na...@gmail.com>
> *Cc: *"user @spark" <us...@spark.apache.org>
> *Subject: *Re: Lost executor 4 Container killed by YARN for exceeding
> memory limits.
>
>
>
> Setting Spark's memoryOverhead configuration variable is recommended in
> your logs, and has helped me with these issues in the past. Search for
> "memoryOverhead" here: http://spark.apache.org/docs/
> latest/running-on-yarn.html
>
>
>
> That said, you're running on a huge cluster as it is. If it's possible to
> filter your tables down before the join (keeping just the rows/columns you
> need), that may be a better solution.
>
>
>
> Jon
>
>
>
> On Mon, Feb 13, 2017 at 5:27 AM, nancy henry <na...@gmail.com>
> wrote:
>
> Hi All,,
>
>
>
> I am getting below error while I am trying to join 3 tables which are in
> ORC format in hive from 5 10gb tables through hive context in spark
>
>
>
> Container killed by YARN for exceeding memory limits. 11.1 GB of 11 GB
> physical memory used. Consider boosting spark.yarn.executor.
> memoryOverhead.
>
> 17/02/13 02:21:19 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:
> Container killed by YARN for exceeding memory limits. 11.1 GB of 11 GB
> physical memory used
>
>
>
>
>
> I am using below memory parameters to launch shell .. what else i could
> increase from these parameters or do I need to change any configuration
> settings please let me know
>
>
>
> spark-shell --master yarn --deploy-mode client --driver-memory 16G
> --num-executors 500 executor-cores 7 --executor-memory 10G
>
>
>
>
>
Re: Lost executor 4 Container killed by YARN for exceeding memory
limits.
Posted by "Thakrar, Jayesh" <jt...@conversantmedia.com>.
Nancy,
As your log output indicated, your executor 11 GB memory limit.
While you might want to address the root cause/data volume as suggested by Jon, you can do an immediate test by changing your command as follows
spark-shell --master yarn --deploy-mode client --driver-memory 16G --num-executors 500 executor-cores 7 --executor-memory 14G
This essentially increases your executor memory from 11 GB to 14 GB.
Note that it will result in a potentially large footprint - from 500x11 to 500x14 GB.
You may want to consult with your DevOps/Operations/Spark Admin team first.
From: Jon Gregg <co...@gmail.com>
Date: Monday, February 13, 2017 at 8:58 AM
To: nancy henry <na...@gmail.com>
Cc: "user @spark" <us...@spark.apache.org>
Subject: Re: Lost executor 4 Container killed by YARN for exceeding memory limits.
Setting Spark's memoryOverhead configuration variable is recommended in your logs, and has helped me with these issues in the past. Search for "memoryOverhead" here: http://spark.apache.org/docs/latest/running-on-yarn.html
That said, you're running on a huge cluster as it is. If it's possible to filter your tables down before the join (keeping just the rows/columns you need), that may be a better solution.
Jon
On Mon, Feb 13, 2017 at 5:27 AM, nancy henry <na...@gmail.com>> wrote:
Hi All,,
I am getting below error while I am trying to join 3 tables which are in ORC format in hive from 5 10gb tables through hive context in spark
Container killed by YARN for exceeding memory limits. 11.1 GB of 11 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
17/02/13 02:21:19 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 11.1 GB of 11 GB physical memory used
I am using below memory parameters to launch shell .. what else i could increase from these parameters or do I need to change any configuration settings please let me know
spark-shell --master yarn --deploy-mode client --driver-memory 16G --num-executors 500 executor-cores 7 --executor-memory 10G
Re: Lost executor 4 Container killed by YARN for exceeding memory limits.
Posted by Jon Gregg <co...@gmail.com>.
Setting Spark's memoryOverhead configuration variable is recommended in
your logs, and has helped me with these issues in the past. Search for
"memoryOverhead" here:
http://spark.apache.org/docs/latest/running-on-yarn.html
That said, you're running on a huge cluster as it is. If it's possible to
filter your tables down before the join (keeping just the rows/columns you
need), that may be a better solution.
Jon
On Mon, Feb 13, 2017 at 5:27 AM, nancy henry <na...@gmail.com>
wrote:
> Hi All,,
>
> I am getting below error while I am trying to join 3 tables which are in
> ORC format in hive from 5 10gb tables through hive context in spark
>
> Container killed by YARN for exceeding memory limits. 11.1 GB of 11 GB
> physical memory used. Consider boosting spark.yarn.executor.
> memoryOverhead.
> 17/02/13 02:21:19 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:
> Container killed by YARN for exceeding memory limits. 11.1 GB of 11 GB
> physical memory used
>
>
> I am using below memory parameters to launch shell .. what else i could
> increase from these parameters or do I need to change any configuration
> settings please let me know
>
> spark-shell --master yarn --deploy-mode client --driver-memory 16G
> --num-executors 500 executor-cores 7 --executor-memory 10G
>
>