You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2019/06/09 23:50:00 UTC
[jira] [Commented] (SPARK-26988) Spark overwrites
spark.scheduler.pool if set in configs
[ https://issues.apache.org/jira/browse/SPARK-26988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16859615#comment-16859615 ]
Dongjoon Hyun commented on SPARK-26988:
---------------------------------------
If there is no problem when we use `--conf` option or load from the file, what about updating the doc instead?
> Spark overwrites spark.scheduler.pool if set in configs
> -------------------------------------------------------
>
> Key: SPARK-26988
> URL: https://issues.apache.org/jira/browse/SPARK-26988
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler
> Affects Versions: 2.4.0
> Reporter: Dave DeCaprio
> Priority: Minor
>
> If you set a default spark.scheduler.pool in your configuration when you create a SparkSession and then you attempt to override that configuration by calling setLocalProperty on a SparkSession, as described in the Spark documentation - [https://spark.apache.org/docs/latest/job-scheduling.html#fair-scheduler-pools] - it won't work.
> Spark will go with the original pool name.
> I've traced this down to SQLExecution.withSQLConfPropagated, which copies any key that starts with "spark" from the the session state to the local properties. The can end up overwriting the scheduler, which is set by spark.scheduler.pool
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org