You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by rdblue <gi...@git.apache.org> on 2016/03/04 23:19:07 UTC
[GitHub] spark pull request: SPARK-13688: Add spark.dynamicAllocation.overr...
GitHub user rdblue opened a pull request:
https://github.com/apache/spark/pull/11528
SPARK-13688: Add spark.dynamicAllocation.overrideNumInstances.
When both the number of executor instances and dynamic allocation are configured, dynamic allocation is ignored and the number of executors is statically set. This introduces a configuration property to change this behavior so that cluster administrators can make dynamic allocation the default when users set `--num-executors` instead of `spark.dynamicAllocation.minExecutors` and unintentionally disable dynamic allocation for a job.
This includes unit tests for the new cases and updates to documentation.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rdblue/spark SPARK-13688-override-num-instances
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11528.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11528
----
commit e79fe20dee37946a188e1b86578fa1f0872b39e6
Author: Ryan Blue <bl...@apache.org>
Date: 2016-03-04T21:44:13Z
SPARK-13688: Add spark.dynamicAllocation.overrideNumInstances.
When both the number of executor instances and dynamic allocation are
configured, dynamic allocation is ignored and the number of executors is
statically set. This introduces a configuration property to change this
behavior so that cluster administrators can make dynamic allocation the
default when users set --num-executors instead of
spark.dynamicAllocation.minExecutors and unintentionally disable dynamic
allocation for a job.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13688: Add spark.dynamicAllocation.overr...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11528#issuecomment-193309137
I agree with the above that we shouldn't add another config for this. I would rather see the default behavior changed in the 2.x line. If dynamic allocation config is on then num-executors goes to max and initial # of executors. I think this would allow users to easily cap their usage and would still allow it to free up executors. It would also allow users doing ML start out with a # of executors and if they are actually caching the data the executors wouldn't be freed up. So you would get very similar behavior to if dynamic allocation was off.
thoughts?
We have been running into this issue a lot recently on our clusters where we see a lot of wasted resources and so the dynamic allocation would help but users are so used to specifying --num-executors they don't end up using it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13688: Add spark.dynamicAllocation.overr...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:
https://github.com/apache/spark/pull/11528#issuecomment-192544174
@andrewor14, thanks for taking a look at this so quickly.
From what we've seen, what you suggest isn't a viable work-around in practice because the two properties are set at different times and aren't obviously related. We default jobs to dynamic allocation, so that's an admin configuration most users don't see (unless they want to opt in for static). Users set up their applications and it isn't clear to at that time that requesting some number of executors results in static allocation. They also don't necessarily see the notification in their job logs.
I think a reasonable interpretation is that --num-executors would control the minimum or the initial number of executors. It's fine with me that the default goes the other way, but we lose a lot of resources to accidental static allocation. I think this additional option is worth the trade-off, but I'm happy to discuss other approaches. The main problem with your suggestion is just that users aren't aware that the two are related.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13688: Add spark.dynamicAllocation.overr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11528#issuecomment-192497299
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13688: Add spark.dynamicAllocation.overr...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue closed the pull request at:
https://github.com/apache/spark/pull/11528
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13688: Add spark.dynamicAllocation.overr...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11528#issuecomment-193347167
filed https://issues.apache.org/jira/browse/SPARK-13723 for this so we can move discussion there
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13688: Add spark.dynamicAllocation.overr...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/11528#issuecomment-192502690
I am against this change. We're creating a flag to control the behavior of two other flags. This level of complexity is difficult to maintain. We've already established the precedence order in SPARK-9092 so we shouldn't try to change that. Besides the simple workaround is just to not set one of the configs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13688: Add spark.dynamicAllocation.overr...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/11528#issuecomment-202632874
OK, either way I think we've decided we're not going to proceed with the changes in this patch. @rdblue would you mind closing this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13688: Add spark.dynamicAllocation.overr...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/11528#issuecomment-192608572
Letting `--num-executors` control initial executors is more reasonable to me. We unfortunately already have a different flag for that, which makes it kind of funny. It also only controls initial executors; someone who thinks "this job needs more resource" really wants to up the max, or not enable dynamic allocation at all.
I agree it's an issue for something like CDH that sets dynamic allocation by default, which is a good default. I still think the best solution is docs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13688: Add spark.dynamicAllocation.overr...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:
https://github.com/apache/spark/pull/11528#issuecomment-203531625
Close
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13688: Add spark.dynamicAllocation.overr...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11528#discussion_r55118075
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2253,13 +2253,38 @@ private[spark] object Utils extends Logging {
def isDynamicAllocationEnabled(conf: SparkConf): Boolean = {
val numExecutor = conf.getInt("spark.executor.instances", 0)
val dynamicAllocationEnabled = conf.getBoolean("spark.dynamicAllocation.enabled", false)
+ val overrideExecutors = conf.getBoolean("spark.dynamicAllocation.overrideNumInstances", false)
--- End diff --
I think this just piles onto the confusion though, with yet another flag to control flags. I think this is a documentation issue, and it already generates warnings when you set these things together. I personally favor leaving it as is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org