You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Andy Davidson <An...@SantaCruzIntegration.com> on 2015/11/03 01:04:13 UTC

ipython notebook NameError: name 'sc' is not defined

Hi

I recently installed a new cluster using the
spark-1.5.1-bin-hadoop2.6/ec2/spark-ec2. SparkPi sample app works correctly.

I am trying to run iPython notebook on my cluster master and use an ssh
tunnel so that I can work with the notebook in a browser running on my mac.
Bellow is how I set up the ssh tunnel

$ ssh -i $KEY_FILE -N -f -L localhost:8888:localhost:7000
ec2-user@$SPARK_MASTER

$ ssh -i $KEY_FILE ec2-user@$SPARK_MASTER
$ cd top level notebook dir
$ IPYTHON_OPTS="notebook --no-browser --port=7000" /root/spark/bin/pyspark

I am able to access my notebooks in the browser by opening
http://localhost:8888

When I run the following python code I get an error NameError: name 'sc' is
not defined? Any idea what the problem might be?

I looked through pyspark and tried various combinations of the following but
still get the same error

$ PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook
--no-browser --port=7000" /root/spark/bin/pyspark --master=local[2]

Kind regards

Andy





In [1]:
import sys
print (sys.version)
 
import os
print(os.getcwd() + "\n")
2.6.9 (unknown, Apr  1 2015, 18:16:00)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-16)]
/home/ec2-user/dataScience

In [2]:
from pyspark import SparkContext
textFile = sc.textFile("readme.txt")
textFile.take(1)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-2-b67a9be29bd9> in <module>()
      1 from pyspark import SparkContext
----> 2 textFile = sc.textFile("readme.txt")
      3 textFile.take(1)

NameError: name 'sc' is not defined

In [ ]:
 



Re: ipython notebook NameError: name 'sc' is not defined

Posted by Jörn Franke <jo...@gmail.com>.
You can check a script that I created for the Amazon cloud:
https://snippetessay.wordpress.com/2015/04/18/big-data-lab-in-the-cloud-with-hadoopsparkrpython/

If I remember correctly then you need to add something to the startup py for ipython
> On 03 Nov 2015, at 01:04, Andy Davidson <An...@SantaCruzIntegration.com> wrote:
> 
> Hi
> 
> I recently installed a new cluster using the spark-1.5.1-bin-hadoop2.6/ec2/spark-ec2. SparkPi sample app works correctly. 
> 
> I am trying to run iPython notebook on my cluster master and use an ssh tunnel so that I can work with the notebook in a browser running on my mac. Bellow is how I set up the ssh tunnel
> 
> 	$ ssh -i $KEY_FILE -N -f -L localhost:8888:localhost:7000 ec2-user@$SPARK_MASTER
> 
> 	$ ssh -i $KEY_FILE ec2-user@$SPARK_MASTER
> 	$ cd top level notebook dir
> 	$ IPYTHON_OPTS="notebook --no-browser --port=7000" /root/spark/bin/pyspark
> 
> I am able to access my notebooks in the browser by opening http://localhost:8888
> 
> When I run the following python code I get an error NameError: name 'sc' is not defined? Any idea what the problem might be? 
> 
> I looked through pyspark and tried various combinations of the following but still get the same error
> 
> $ PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook --no-browser --port=7000" /root/spark/bin/pyspark --master=local[2]
> 
> Kind regards
> 
> Andy
> 
> 
> 
> 
> 
> In [1]:
> 
> import sys
> print (sys.version)
>  
> import os
> print(os.getcwd() + "\n")
> 2.6.9 (unknown, Apr  1 2015, 18:16:00) 
> [GCC 4.8.2 20140120 (Red Hat 4.8.2-16)]
> /home/ec2-user/dataScience
> 
> In [2]:
> 
> from pyspark import SparkContext
> textFile = sc.textFile("readme.txt")
> textFile.take(1)
> ---------------------------------------------------------------------------
> NameError                                 Traceback (most recent call last)
> <ipython-input-2-b67a9be29bd9> in <module>()
>       1 from pyspark import SparkContext
> ----> 2 textFile = sc.textFile("readme.txt")
>       3 textFile.take(1)
> 
> NameError: name 'sc' is not defined
> 
> In [ ]:
> 
>