You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by an...@apache.org on 2015/06/10 22:35:58 UTC
spark git commit: [SPARK-8273] Driver hangs up when yarn shutdown in
client mode
Repository: spark
Updated Branches:
refs/heads/master cb871c44c -> 5014d0ed7
[SPARK-8273] Driver hangs up when yarn shutdown in client mode
In client mode, if yarn was shut down with spark application running, the application will hang up after several retries(default: 30) because the exception throwed by YarnClientImpl could not be caught by upper level, we should exit in case that user can not be aware that.
The exception we wanna catch is [here](https://github.com/apache/hadoop/blob/branch-2.7.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java#L122), and I try to fix it refer to [MR](https://github.com/apache/hadoop/blob/branch-2.7.0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java#L320).
Author: WangTaoTheTonic <wa...@huawei.com>
Closes #6717 from WangTaoTheTonic/SPARK-8273 and squashes the following commits:
28752d6 [WangTaoTheTonic] catch the throwed exception
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5014d0ed
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5014d0ed
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5014d0ed
Branch: refs/heads/master
Commit: 5014d0ed7e2f69810654003f8dd38078b945cf05
Parents: cb871c4
Author: WangTaoTheTonic <wa...@huawei.com>
Authored: Wed Jun 10 13:34:19 2015 -0700
Committer: Andrew Or <an...@databricks.com>
Committed: Wed Jun 10 13:35:51 2015 -0700
----------------------------------------------------------------------
yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala | 4 ++++
1 file changed, 4 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/5014d0ed/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
----------------------------------------------------------------------
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
index ec9402a..da1ec2a 100644
--- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
+++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
@@ -29,6 +29,7 @@ import scala.collection.JavaConversions._
import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet, ListBuffer, Map}
import scala.reflect.runtime.universe
import scala.util.{Try, Success, Failure}
+import scala.util.control.NonFatal
import com.google.common.base.Charsets.UTF_8
import com.google.common.base.Objects
@@ -826,6 +827,9 @@ private[spark] class Client(
case e: ApplicationNotFoundException =>
logError(s"Application $appId not found.")
return (YarnApplicationState.KILLED, FinalApplicationStatus.KILLED)
+ case NonFatal(e) =>
+ logError(s"Failed to contact YARN for application $appId.", e)
+ return (YarnApplicationState.FAILED, FinalApplicationStatus.FAILED)
}
val state = report.getYarnApplicationState
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org