You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by da...@apache.org on 2016/05/25 21:29:18 UTC
spark git commit: [MINOR] [PYSPARK] [EXAMPLES] Changed examples to
use SparkSession.sparkContext instead of _sc
Repository: spark
Updated Branches:
refs/heads/master 698ef762f -> 9c297df3d
[MINOR] [PYSPARK] [EXAMPLES] Changed examples to use SparkSession.sparkContext instead of _sc
## What changes were proposed in this pull request?
Some PySpark examples need a SparkContext and get it by accessing _sc directly from the session. These examples should use the provided property `sparkContext` in `SparkSession` instead.
## How was this patch tested?
Ran modified examples
Author: Bryan Cutler <cu...@gmail.com>
Closes #13303 from BryanCutler/pyspark-session-sparkContext-MINOR.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9c297df3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9c297df3
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9c297df3
Branch: refs/heads/master
Commit: 9c297df3d4d5fa4bbfdffdaad15f362586db384b
Parents: 698ef76
Author: Bryan Cutler <cu...@gmail.com>
Authored: Wed May 25 14:29:14 2016 -0700
Committer: Davies Liu <da...@gmail.com>
Committed: Wed May 25 14:29:14 2016 -0700
----------------------------------------------------------------------
examples/src/main/python/als.py | 2 +-
examples/src/main/python/avro_inputformat.py | 2 +-
examples/src/main/python/parquet_inputformat.py | 2 +-
examples/src/main/python/pi.py | 2 +-
examples/src/main/python/transitive_closure.py | 2 +-
5 files changed, 5 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/9c297df3/examples/src/main/python/als.py
----------------------------------------------------------------------
diff --git a/examples/src/main/python/als.py b/examples/src/main/python/als.py
index 81562e2..80290e7 100755
--- a/examples/src/main/python/als.py
+++ b/examples/src/main/python/als.py
@@ -67,7 +67,7 @@ if __name__ == "__main__":
.appName("PythonALS")\
.getOrCreate()
- sc = spark._sc
+ sc = spark.sparkContext
M = int(sys.argv[1]) if len(sys.argv) > 1 else 100
U = int(sys.argv[2]) if len(sys.argv) > 2 else 500
http://git-wip-us.apache.org/repos/asf/spark/blob/9c297df3/examples/src/main/python/avro_inputformat.py
----------------------------------------------------------------------
diff --git a/examples/src/main/python/avro_inputformat.py b/examples/src/main/python/avro_inputformat.py
index 3f65e8f..4422f9e 100644
--- a/examples/src/main/python/avro_inputformat.py
+++ b/examples/src/main/python/avro_inputformat.py
@@ -70,7 +70,7 @@ if __name__ == "__main__":
.appName("AvroKeyInputFormat")\
.getOrCreate()
- sc = spark._sc
+ sc = spark.sparkContext
conf = None
if len(sys.argv) == 3:
http://git-wip-us.apache.org/repos/asf/spark/blob/9c297df3/examples/src/main/python/parquet_inputformat.py
----------------------------------------------------------------------
diff --git a/examples/src/main/python/parquet_inputformat.py b/examples/src/main/python/parquet_inputformat.py
index 2f09f4d..29a1ac2 100644
--- a/examples/src/main/python/parquet_inputformat.py
+++ b/examples/src/main/python/parquet_inputformat.py
@@ -53,7 +53,7 @@ if __name__ == "__main__":
.appName("ParquetInputFormat")\
.getOrCreate()
- sc = spark._sc
+ sc = spark.sparkContext
parquet_rdd = sc.newAPIHadoopFile(
path,
http://git-wip-us.apache.org/repos/asf/spark/blob/9c297df3/examples/src/main/python/pi.py
----------------------------------------------------------------------
diff --git a/examples/src/main/python/pi.py b/examples/src/main/python/pi.py
index 5db03e4..b39d710 100755
--- a/examples/src/main/python/pi.py
+++ b/examples/src/main/python/pi.py
@@ -32,7 +32,7 @@ if __name__ == "__main__":
.appName("PythonPi")\
.getOrCreate()
- sc = spark._sc
+ sc = spark.sparkContext
partitions = int(sys.argv[1]) if len(sys.argv) > 1 else 2
n = 100000 * partitions
http://git-wip-us.apache.org/repos/asf/spark/blob/9c297df3/examples/src/main/python/transitive_closure.py
----------------------------------------------------------------------
diff --git a/examples/src/main/python/transitive_closure.py b/examples/src/main/python/transitive_closure.py
index 37c41dc..d88ea94 100755
--- a/examples/src/main/python/transitive_closure.py
+++ b/examples/src/main/python/transitive_closure.py
@@ -46,7 +46,7 @@ if __name__ == "__main__":
.appName("PythonTransitiveClosure")\
.getOrCreate()
- sc = spark._sc
+ sc = spark.sparkContext
partitions = int(sys.argv[1]) if len(sys.argv) > 1 else 2
tc = sc.parallelize(generateGraph(), partitions).cache()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org