You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by an...@apache.org on 2014/11/19 21:11:13 UTC
spark git commit: [SPARK-4470] Validate number of threads in local
mode
Repository: spark
Updated Branches:
refs/heads/master d75579d09 -> eacc78834
[SPARK-4470] Validate number of threads in local mode
When running Spark locally, if number of threads is specified as 0 (e.g., `spark-submit --master local[0] ...`), the job got stuck and does not run at all.
I think it's better to validate the parameter.
Fix for [SPARK-4470](https://issues.apache.org/jira/browse/SPARK-4470).
Author: Kenichi Maehashi <we...@kenichimaehashi.com>
Closes #3337 from kmaehashi/spark-4470 and squashes the following commits:
3ad76f3 [Kenichi Maehashi] fix code style
7716734 [Kenichi Maehashi] SPARK-4470: Validate number of threads in local mode
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/eacc7883
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/eacc7883
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/eacc7883
Branch: refs/heads/master
Commit: eacc788346ccae232bd530dd880f801475a49734
Parents: d75579d
Author: Kenichi Maehashi <we...@kenichimaehashi.com>
Authored: Wed Nov 19 12:11:09 2014 -0800
Committer: Andrew Or <an...@databricks.com>
Committed: Wed Nov 19 12:11:09 2014 -0800
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/SparkContext.scala | 3 +++
1 file changed, 3 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/eacc7883/core/src/main/scala/org/apache/spark/SparkContext.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 3701312..ae8bbfb 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -1813,6 +1813,9 @@ object SparkContext extends Logging {
def localCpuCount = Runtime.getRuntime.availableProcessors()
// local[*] estimates the number of cores on the machine; local[N] uses exactly N threads.
val threadCount = if (threads == "*") localCpuCount else threads.toInt
+ if (threadCount <= 0) {
+ throw new SparkException(s"Asked to run locally with $threadCount threads")
+ }
val scheduler = new TaskSchedulerImpl(sc, MAX_LOCAL_TASK_FAILURES, isLocal = true)
val backend = new LocalBackend(scheduler, threadCount)
scheduler.initialize(backend)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org