You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by viirya <gi...@git.apache.org> on 2015/12/11 07:25:10 UTC
[GitHub] spark pull request: [SPARK-12276][Core] Prevent RejectedExecutionE...
GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/10267
[SPARK-12276][Core] Prevent RejectedExecutionException by checking if ThreadPoolExecutor is shutdown and its capacity
JIRA: https://issues.apache.org/jira/browse/SPARK-12276
We noticed that it is possible to throw RejectedExecutionException when submitting thread in AppClient. The error is like following. We should add some checks to prevent it.
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2077082c rejected from java.util.concurrent.ThreadPoolExecutor@66b9915a[Running, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(AppClient.scala:96)
...
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 fix-reject-execution-exception
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10267.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10267
----
commit d8fc88d19fd68b3cb9b34f34a5a624fe26b31c12
Author: Liang-Chi Hsieh <vi...@appier.com>
Date: 2015-12-11T06:18:53Z
Prevent RejectedExecutionException by checking if ThreadPoolExecutor is shutdown and its capacity.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12276][Core] Prevent RejectedExecutionE...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10267#issuecomment-164092733
Yes, I forgot about that patch. This should not be a problem anymore, and this isn't the correct fix even if it is. @viirya would you mind closing this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12276][Core] Prevent RejectedExecutionE...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10267#discussion_r47409546
--- Diff: core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala ---
@@ -96,22 +96,27 @@ private[spark] class AppClient(
/**
* Register with all masters asynchronously and returns an array `Future`s for cancellation.
*/
- private def tryRegisterAllMasters(): Array[JFuture[_]] = {
+ private def tryRegisterAllMasters(): Array[Option[JFuture[_]]] = {
--- End diff --
you don't have to change the method signature here if you just to a flatten within this method
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12276][Core] Prevent RejectedExecutionE...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10267#issuecomment-163871741
**[Test build #47575 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47575/consoleFull)** for PR 10267 at commit [`d8fc88d`](https://github.com/apache/spark/commit/d8fc88d19fd68b3cb9b34f34a5a624fe26b31c12).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12276][Core] Prevent RejectedExecutionE...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10267#issuecomment-163855617
**[Test build #47575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47575/consoleFull)** for PR 10267 at commit [`d8fc88d`](https://github.com/apache/spark/commit/d8fc88d19fd68b3cb9b34f34a5a624fe26b31c12).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12276][Core] Prevent RejectedExecutionE...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10267#issuecomment-164100820
I am not sure if that patch solves this problem. As there is still the bug [reported](http://stackoverflow.com/questions/34065209/java-util-concurrent-rejectedexecutionexception-in-spark-although-driver-client). Maybe it is just before the patch is committed. I would close this first. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12276][Core] Prevent RejectedExecutionE...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10267#issuecomment-163871828
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47575/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12276][Core] Prevent RejectedExecutionE...
Posted by viirya <gi...@git.apache.org>.
Github user viirya closed the pull request at:
https://github.com/apache/spark/pull/10267
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12276][Core] Prevent RejectedExecutionE...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/10267#issuecomment-164062015
I think this is already fixed by #10108. Could you try the master or 1.6 branch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12276][Core] Prevent RejectedExecutionE...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10267#discussion_r47409560
--- Diff: core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala ---
@@ -123,7 +128,7 @@ private[spark] class AppClient(
* nthRetry means this is the nth attempt to register with master.
*/
private def registerWithMaster(nthRetry: Int) {
- registerMasterFutures.set(tryRegisterAllMasters())
+ registerMasterFutures.set(tryRegisterAllMasters().flatMap(x => x))
--- End diff --
can be just `.flatten`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12276][Core] Prevent RejectedExecutionE...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10267#issuecomment-164059133
@viirya @zsxwing what is the root cause of this issue? It seems arbitrary to just cut off an attempt if the pool size is already at the capacity. I would think that we should still make the attempt, but how can we exceed the thread pool capacity if our pool size is already `masterRpcAddresses.size`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12276][Core] Prevent RejectedExecutionE...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10267#issuecomment-163871826
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org