You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2018/08/24 13:59:00 UTC
[jira] [Resolved] (SPARK-25073) Spark-submit on Yarn Task : When
the yarn.nodemanager.resource.memory-mb and/or
yarn.scheduler.maximum-allocation-mb is insufficient, Spark always reports
an error request to adjust yarn.scheduler.maximum-allocation-mb
[ https://issues.apache.org/jira/browse/SPARK-25073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-25073.
-------------------------------
Resolution: Fixed
Fix Version/s: 2.4.0
Issue resolved by pull request 22199
[https://github.com/apache/spark/pull/22199]
> Spark-submit on Yarn Task : When the yarn.nodemanager.resource.memory-mb and/or yarn.scheduler.maximum-allocation-mb is insufficient, Spark always reports an error request to adjust yarn.scheduler.maximum-allocation-mb
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-25073
> URL: https://issues.apache.org/jira/browse/SPARK-25073
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
> Affects Versions: 2.3.0, 2.3.1
> Reporter: vivek kumar
> Assignee: Sujith
> Priority: Trivial
> Fix For: 2.4.0
>
>
> When the yarn.nodemanager.resource.memory-mb and/or yarn.scheduler.maximum-allocation-mb is insufficient, Spark *always* reports an error request to adjust Yarn.scheduler.maximum-allocation-mb. Expecting the error request to be more around yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.
>
> Scenario 1. yarn.scheduler.maximum-allocation-mb =4g and yarn.nodemanager.resource.memory-mb =8G
> # Launch shell on Yarn with am.memory less than nodemanager.resource memory but greater than yarn.scheduler.maximum-allocation-mb
> eg; spark-shell --master yarn --conf spark.yarn.am.memory 5g
> Error: java.lang.IllegalArgumentException: Required AM memory (5120+512 MB) is above the max threshold (4096 MB) of this cluster! Please increase the value of 'yarn.scheduler.maximum-allocation-mb'.
> at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:325)
>
> *Scenario 2*. yarn.scheduler.maximum-allocation-mb =15g and yarn.nodemanager.resource.memory-mb =8g
> a. Launch shell on Yarn with am.memory greater than nodemanager.resource memory but less than yarn.scheduler.maximum-allocation-mb
> eg; *spark-shell --master yarn --conf spark.yarn.am.memory=10g*
> Error :
> java.lang.IllegalArgumentException: Required AM memory (10240+1024 MB) is above the max threshold (*8096 MB*) of this cluster! *Please increase the value of 'yarn.scheduler.maximum-allocation-mb'.*
> at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:325)
>
> b. Launch shell on Yarn with am.memory greater than nodemanager.resource memory and yarn.scheduler.maximum-allocation-mb
> eg; *spark-shell --master yarn --conf spark.yarn.am.memory=17g*
> Error:
> java.lang.IllegalArgumentException: Required AM memory (17408+1740 MB) is above the max threshold (*8096 MB*) of this cluster! *Please increase the value of 'yarn.scheduler.maximum-allocation-mb'.*
> at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:325)
>
> *Expected* : Error request for scenario2 should be more around yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org