You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Andy Davidson <An...@SantaCruzIntegration.com> on 2015/11/03 01:04:13 UTC
ipython notebook NameError: name 'sc' is not defined
Hi
I recently installed a new cluster using the
spark-1.5.1-bin-hadoop2.6/ec2/spark-ec2. SparkPi sample app works correctly.
I am trying to run iPython notebook on my cluster master and use an ssh
tunnel so that I can work with the notebook in a browser running on my mac.
Bellow is how I set up the ssh tunnel
$ ssh -i $KEY_FILE -N -f -L localhost:8888:localhost:7000
ec2-user@$SPARK_MASTER
$ ssh -i $KEY_FILE ec2-user@$SPARK_MASTER
$ cd top level notebook dir
$ IPYTHON_OPTS="notebook --no-browser --port=7000" /root/spark/bin/pyspark
I am able to access my notebooks in the browser by opening
http://localhost:8888
When I run the following python code I get an error NameError: name 'sc' is
not defined? Any idea what the problem might be?
I looked through pyspark and tried various combinations of the following but
still get the same error
$ PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook
--no-browser --port=7000" /root/spark/bin/pyspark --master=local[2]
Kind regards
Andy
In [1]:
import sys
print (sys.version)
import os
print(os.getcwd() + "\n")
2.6.9 (unknown, Apr 1 2015, 18:16:00)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-16)]
/home/ec2-user/dataScience
In [2]:
from pyspark import SparkContext
textFile = sc.textFile("readme.txt")
textFile.take(1)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-2-b67a9be29bd9> in <module>()
1 from pyspark import SparkContext
----> 2 textFile = sc.textFile("readme.txt")
3 textFile.take(1)
NameError: name 'sc' is not defined
In [ ]:
Re: ipython notebook NameError: name 'sc' is not defined
Posted by Jörn Franke <jo...@gmail.com>.
You can check a script that I created for the Amazon cloud:
https://snippetessay.wordpress.com/2015/04/18/big-data-lab-in-the-cloud-with-hadoopsparkrpython/
If I remember correctly then you need to add something to the startup py for ipython
> On 03 Nov 2015, at 01:04, Andy Davidson <An...@SantaCruzIntegration.com> wrote:
>
> Hi
>
> I recently installed a new cluster using the spark-1.5.1-bin-hadoop2.6/ec2/spark-ec2. SparkPi sample app works correctly.
>
> I am trying to run iPython notebook on my cluster master and use an ssh tunnel so that I can work with the notebook in a browser running on my mac. Bellow is how I set up the ssh tunnel
>
> $ ssh -i $KEY_FILE -N -f -L localhost:8888:localhost:7000 ec2-user@$SPARK_MASTER
>
> $ ssh -i $KEY_FILE ec2-user@$SPARK_MASTER
> $ cd top level notebook dir
> $ IPYTHON_OPTS="notebook --no-browser --port=7000" /root/spark/bin/pyspark
>
> I am able to access my notebooks in the browser by opening http://localhost:8888
>
> When I run the following python code I get an error NameError: name 'sc' is not defined? Any idea what the problem might be?
>
> I looked through pyspark and tried various combinations of the following but still get the same error
>
> $ PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook --no-browser --port=7000" /root/spark/bin/pyspark --master=local[2]
>
> Kind regards
>
> Andy
>
>
>
>
>
> In [1]:
>
> import sys
> print (sys.version)
>
> import os
> print(os.getcwd() + "\n")
> 2.6.9 (unknown, Apr 1 2015, 18:16:00)
> [GCC 4.8.2 20140120 (Red Hat 4.8.2-16)]
> /home/ec2-user/dataScience
>
> In [2]:
>
> from pyspark import SparkContext
> textFile = sc.textFile("readme.txt")
> textFile.take(1)
> ---------------------------------------------------------------------------
> NameError Traceback (most recent call last)
> <ipython-input-2-b67a9be29bd9> in <module>()
> 1 from pyspark import SparkContext
> ----> 2 textFile = sc.textFile("readme.txt")
> 3 textFile.take(1)
>
> NameError: name 'sc' is not defined
>
> In [ ]:
>
>