You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dhruve <gi...@git.apache.org> on 2018/08/06 20:44:31 UTC
[GitHub] spark pull request #22015: [SPARK-20286] Release executors on unpersisting R...
GitHub user dhruve opened a pull request:
https://github.com/apache/spark/pull/22015
[SPARK-20286] Release executors on unpersisting RDD
## What changes were proposed in this pull request?
Currently, the executors acquired using dynamic allocation are not released when the cached RDD is unpersisted. This leads to wasting unnecessary grid resources. With this change, once the cached RDD is unpersisted, we check if the executor has any running tasks or not. If not then we do the following:
1 - If the executor has cached RDD blocks from other RDDs, we don't make any change.
2 - If the executor has no more cached RDD blocks and tasks running, we update the removal time based on the conf `spark.dynamicAllocation.cachedExecutorIdleTimeout` so the idle executor can be released back.
## How was this patch tested?
Manually using a code snippet.
``` scala
val rdd = sc.textFile("smallFile")
rdd.cache
val rdd2 = sc.textFile("largeFile")
rdd2.cache
rdd2.count // Cached data on around 500+ executors
Thread.sleep(30000) // sleep for 30s
rdd.count // Cached data on around 20 executors
// Verify only 20 executors remain, rest will timeout based on idleTimeout which i set to 60s
rdd2.unpersist
// eventunally all executors will be released as there are no tasks running on any executor.
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dhruve/spark bug/SPARK-20286
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22015.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22015
----
commit 7e229dc120d0f1542cc8d7dbac1027baac36665e
Author: Dhruve Ashar <dh...@...>
Date: 2018-08-06T20:32:47Z
[SPARK-20286] Release executors on unpersisting RDD
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Rele...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22015
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1872/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Rele...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22015
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Rele...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22015
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Rele...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22015
**[Test build #94316 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94316/testReport)** for PR 22015 at commit [`7e229dc`](https://github.com/apache/spark/commit/7e229dc120d0f1542cc8d7dbac1027baac36665e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Rele...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22015
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1871/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Rele...
Posted by dhruve <gi...@git.apache.org>.
Github user dhruve commented on the issue:
https://github.com/apache/spark/pull/22015
This modifies the SparkListenerUnpersistRDD case class, which fails the MiMa tests. If this is something heavily used by developers, I can add either another ListenerBus message or modify the PR to make a direct RPC call to the ExecutorAllocationManager.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Rele...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22015
**[Test build #94317 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94317/testReport)** for PR 22015 at commit [`fdb27df`](https://github.com/apache/spark/commit/fdb27df5eb567f98748d3fafa797652b50c11737).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Rele...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22015
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Rele...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22015
**[Test build #94316 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94316/testReport)** for PR 22015 at commit [`7e229dc`](https://github.com/apache/spark/commit/7e229dc120d0f1542cc8d7dbac1027baac36665e).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class SparkListenerUnpersistRDD(rddId: Int, executorIds: Seq[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Rele...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22015
**[Test build #94317 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94317/testReport)** for PR 22015 at commit [`fdb27df`](https://github.com/apache/spark/commit/fdb27df5eb567f98748d3fafa797652b50c11737).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Rele...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22015
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94316/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Rele...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22015
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94317/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Rele...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22015
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org