You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by rx...@apache.org on 2016/11/29 02:28:27 UTC
spark git commit: [SPARK-18523][PYSPARK] Make SparkContext.stop more
reliable
Repository: spark
Updated Branches:
refs/heads/master 2e809903d -> 71352c94a
[SPARK-18523][PYSPARK] Make SparkContext.stop more reliable
## What changes were proposed in this pull request?
This PR fixes SparkContext broken state in which it may fall if spark driver get crashed or killed by OOM.
## How was this patch tested?
1. Start SparkContext;
2. Find Spark driver process and `kill -9` it;
3. Call `sc.stop()`;
4. Create new SparkContext after that;
Without this patch you will crash on step 3 and won't be able to do step 4 without manual reset private attibutes or IPython notebook / shell restart.
Author: Alexander Shorin <kx...@apache.org>
Closes #15961 from kxepal/18523-make-spark-context-stop-more-reliable.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/71352c94
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/71352c94
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/71352c94
Branch: refs/heads/master
Commit: 71352c94ad2a60d1695bd7ac0f4452539270e10c
Parents: 2e80990
Author: Alexander Shorin <kx...@apache.org>
Authored: Mon Nov 28 18:28:24 2016 -0800
Committer: Reynold Xin <rx...@databricks.com>
Committed: Mon Nov 28 18:28:24 2016 -0800
----------------------------------------------------------------------
python/pyspark/context.py | 17 +++++++++++++++--
1 file changed, 15 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/71352c94/python/pyspark/context.py
----------------------------------------------------------------------
diff --git a/python/pyspark/context.py b/python/pyspark/context.py
index 2fd3aee..5c4e79c 100644
--- a/python/pyspark/context.py
+++ b/python/pyspark/context.py
@@ -26,6 +26,8 @@ import warnings
from threading import RLock
from tempfile import NamedTemporaryFile
+from py4j.protocol import Py4JError
+
from pyspark import accumulators
from pyspark.accumulators import Accumulator
from pyspark.broadcast import Broadcast
@@ -373,8 +375,19 @@ class SparkContext(object):
Shut down the SparkContext.
"""
if getattr(self, "_jsc", None):
- self._jsc.stop()
- self._jsc = None
+ try:
+ self._jsc.stop()
+ except Py4JError:
+ # Case: SPARK-18523
+ warnings.warn(
+ 'Unable to cleanly shutdown Spark JVM process.'
+ ' It is possible that the process has crashed,'
+ ' been killed or may also be in a zombie state.',
+ RuntimeWarning
+ )
+ pass
+ finally:
+ self._jsc = None
if getattr(self, "_accumulatorServer", None):
self._accumulatorServer.shutdown()
self._accumulatorServer = None
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org