You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2016/03/16 10:33:42 UTC
spark git commit: [SPARK-13906] Ensure that there are at least 2
dispatcher threads.
Repository: spark
Updated Branches:
refs/heads/master 431a3d04b -> 05ab2948a
[SPARK-13906] Ensure that there are at least 2 dispatcher threads.
## What changes were proposed in this pull request?
Force at least two dispatcher-event-loop threads. Since SparkDeploySchedulerBackend (in AppClient) calls askWithRetry to CoarseGrainedScheduler in the same process, there the driver needs at least two dispatcher threads to prevent the dispatcher thread from hanging.
## How was this patch tested?
Manual.
Author: Yonathan Randolph <yonathangmail.com>
Author: Yonathan Randolph <yo...@liftigniter.com>
Closes #11728 from yonran/SPARK-13906.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/05ab2948
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/05ab2948
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/05ab2948
Branch: refs/heads/master
Commit: 05ab2948ab357fc07222bb3505df80b1886f7310
Parents: 431a3d0
Author: Yonathan Randolph <yo...@liftigniter.com>
Authored: Wed Mar 16 09:34:04 2016 +0000
Committer: Sean Owen <so...@cloudera.com>
Committed: Wed Mar 16 09:34:04 2016 +0000
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/05ab2948/core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala b/core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala
index f092f1d..613d6ee 100644
--- a/core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala
+++ b/core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala
@@ -192,7 +192,7 @@ private[netty] class Dispatcher(nettyEnv: NettyRpcEnv) extends Logging {
/** Thread pool used for dispatching messages. */
private val threadpool: ThreadPoolExecutor = {
val numThreads = nettyEnv.conf.getInt("spark.rpc.netty.dispatcher.numThreads",
- Runtime.getRuntime.availableProcessors())
+ math.max(2, Runtime.getRuntime.availableProcessors()))
val pool = ThreadUtils.newDaemonFixedThreadPool(numThreads, "dispatcher-event-loop")
for (i <- 0 until numThreads) {
pool.execute(new MessageLoop)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org