You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2016/09/14 08:38:33 UTC
spark git commit: [SPARK-17525][PYTHON] Remove
SparkContext.clearFiles() from the PySpark API as it was removed from the
Scala API prior to Spark 2.0.0
Repository: spark
Updated Branches:
refs/heads/master def7c265f -> b5bfcddbf
[SPARK-17525][PYTHON] Remove SparkContext.clearFiles() from the PySpark API as it was removed from the Scala API prior to Spark 2.0.0
## What changes were proposed in this pull request?
This pull request removes the SparkContext.clearFiles() method from the PySpark API as the method was removed from the Scala API in 8ce645d4eeda203cf5e100c4bdba2d71edd44e6a. Using that method in PySpark leads to an exception as PySpark tries to call the non-existent method on the JVM side.
## How was this patch tested?
Existing tests (though none of them tested this particular method).
Author: Sami Jaktholm <sj...@outlook.com>
Closes #15081 from sjakthol/pyspark-sc-clearfiles.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b5bfcddb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b5bfcddb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b5bfcddb
Branch: refs/heads/master
Commit: b5bfcddbfbc2e79d3d0fbd43942716946e6c4ba3
Parents: def7c26
Author: Sami Jaktholm <sj...@outlook.com>
Authored: Wed Sep 14 09:38:30 2016 +0100
Committer: Sean Owen <so...@cloudera.com>
Committed: Wed Sep 14 09:38:30 2016 +0100
----------------------------------------------------------------------
python/pyspark/context.py | 8 --------
1 file changed, 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/b5bfcddb/python/pyspark/context.py
----------------------------------------------------------------------
diff --git a/python/pyspark/context.py b/python/pyspark/context.py
index 6e9f24e..2744bb9 100644
--- a/python/pyspark/context.py
+++ b/python/pyspark/context.py
@@ -787,14 +787,6 @@ class SparkContext(object):
"""
self._jsc.sc().addFile(path)
- def clearFiles(self):
- """
- Clear the job's list of files added by L{addFile} or L{addPyFile} so
- that they do not get downloaded to any new nodes.
- """
- # TODO: remove added .py or .zip files from the PYTHONPATH?
- self._jsc.sc().clearFiles()
-
def addPyFile(self, path):
"""
Add a .py or .zip dependency for all tasks to be executed on this
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org