You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "shahid (JIRA)" <ji...@apache.org> on 2019/02/27 14:03:00 UTC
[jira] [Commented] (SPARK-27003) [Spark]Message display at console
is not correct for spark.executor.instances
[ https://issues.apache.org/jira/browse/SPARK-27003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779341#comment-16779341 ]
shahid commented on SPARK-27003:
--------------------------------
default value of spark.executor.instances=2 is applicable only for static allocation case. However in case of dynamic allocation, default value is taken as 0. So, this is not as issue.
However documents seems need to be updated, as it is not mentioned whether the default value is applicable only for static allocation.
> [Spark]Message display at console is not correct for spark.executor.instances
> -----------------------------------------------------------------------------
>
> Key: SPARK-27003
> URL: https://issues.apache.org/jira/browse/SPARK-27003
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.0
> Environment: {color:red}colored text{color}
> Reporter: ABHISHEK KUMAR GUPTA
> Priority: Minor
> Attachments: Spark-JIRA-Executor-Msg.png, Spark-JIRA-Executor-Msg.png
>
>
> Steps:
> spark.executor.instances{color:#ff0000} not {color}configured at client side spark-default.conf( Default value is 2 )
> bin/spark-shell --master yarn --executor-memory 1024m --executor-cores 1 --conf spark.dynamicAllocation.enabled=true --conf spark.dynamicAllocation.initialExecutors=3 --conf {color:#ff0000}spark.dynamicAllocation.minExecutors=1{color} --conf spark.dynamicAllocation.executorIdleTimeout=60s --conf spark.authenticate=true
> Warning msg display as below
> !Spark-JIRA-Executor-Msg.png!
> {color:#ff0000}spark.executor.instances less than spark.dynamicAllocation.minExecutors in Invalid, ignoring its setting, please update your config.{color}
>
> Note:
> When I submit as below with {color:#ff0000}spark.dynamicAllocation.minExecutors=0{color} no warning message at console
>
> bin/spark-shell --master yarn --executor-memory 1024m --executor-cores 1 --conf spark.dynamicAllocation.enabled=true --conf spark.dynamicAllocation.initialExecutors=3 --conf spark.dynamicAllocation.minExecutors=0 --conf spark.dynamicAllocation.executorIdleTimeout=60s --conf spark.authenticate=true
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org