You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:25:31 UTC
[jira] [Updated] (SPARK-10702) Dynamic Allocation in Standalone
Breaking Parallelism
[ https://issues.apache.org/jira/browse/SPARK-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-10702:
---------------------------------
Labels: bulk-closed (was: )
> Dynamic Allocation in Standalone Breaking Parallelism
> -----------------------------------------------------
>
> Key: SPARK-10702
> URL: https://issues.apache.org/jira/browse/SPARK-10702
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 1.5.0
> Environment: CentOS 7. Standalone
> Reporter: Mark Khaitman
> Priority: Major
> Labels: bulk-closed
>
> It seems that although executors are properly dropped after they've reached their configured idle timeout setting, even if all cores in the cluster are still available, it is not regaining the full amount back for subsequent spark jobs within that same context.
> For example:
> - A stage has 40 partitions to process and completes successfully. After X seconds, the executors are all expectedly dropped.
> - Then, another stage is set to begin, and plenty of cores and memory are still available on the nodes within the cluster, however, rather than obtaining 40 cores, only 13 got obtained and only 13 active tasks were running.
> - Another concern was that it put all 13 active tasks onto a single node rather than trying to create the usual amount of executors across the cluster (possibly related to this??)
> Not sure of the exact cause of this, though I do know dynamic allocation to the standalone environment is still new so I kind of half-expected some scheduling concerns to possibly come up! Wondering if anyone else has seen this behaviour.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org