You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Andy Davidson <An...@SantaCruzIntegration.com> on 2014/09/26 20:18:38 UTC
problem with spark-ec2 launch script Re: spark-ec2 ERROR: Line magic
function `%matplotlib` not found
Hi Davies
The real issue is about cluster management. I am new to the spark world and
am not a system administrator. It seem like the problem is with the
spark-ec2 launch script. It is installing old version of python
In the mean time I am trying to figure out how I can manually install the
correct version on all the machines in my cluster
Thanks
Andy
From: Davies Liu <da...@databricks.com>
Date: Thursday, September 25, 2014 at 9:58 PM
To: Andrew Davidson <An...@SantaCruzIntegration.com>
Cc: "user@spark.apache.org" <us...@spark.apache.org>
Subject: Re: spark-ec2 ERROR: Line magic function `%matplotlib` not found
> Maybe you have Python 2.7 on master but Python 2.6 in cluster,
> you should upgrade python to 2.7 in cluster, or use python 2.6 in
> master by set PYSPARK_PYTHON=python2.6
>
> On Thu, Sep 25, 2014 at 5:11 PM, Andy Davidson
> <An...@santacruzintegration.com> wrote:
>> Hi
>>
>> I am running into trouble using iPython notebook on my cluster. Use the
>> following command to set the cluster up
>>
>> $ ./spark-ec2 --key-pair=$KEY_PAIR --identity-file=$KEY_FILE
>> --region=$REGION --slaves=$NUM_SLAVES launch $CLUSTER_NAME
>>
>>
>> On master I launch python as follows
>>
>> $ IPYTHON_OPTS="notebook --pylab inline --no-browser --port=7000"
>> $SPARK_HOME/bin/pyspark
>>
>>
>> It looks like the problem is the cluster is using an old version of python
>> and python. Any idea how I can easily upgrade ? The following version works
>> on my mac
>>
>> Thanks
>>
>> Andy
>>
>> {'commit_hash': '681fd77',
>> 'commit_source': 'installation',
>> 'default_encoding': 'UTF-8',
>> 'ipython_path': '/Library/Python/2.7/site-packages/IPython',
>> 'ipython_version': '2.1.0',
>> 'os_name': 'posix',
>> 'platform': 'Darwin-13.3.0-x86_64-i386-64bit',
>> 'sys_executable': '/usr/bin/python',
>> 'sys_platform': 'darwin',
>> 'sys_version': '2.7.5 (default, Mar 9 2014, 22:15:05) \n[GCC 4.2.1
>> Compatible Apple LLVM 5.0 (clang-500.0.68)]¹}
>>
>>
>>
>>
>
Re: problem with spark-ec2 launch script Re: spark-ec2 ERROR: Line
magic function `%matplotlib` not found
Posted by Andy Davidson <An...@SantaCruzIntegration.com>.
Many many thanks
Andy
From: Nicholas Chammas <ni...@gmail.com>
Date: Friday, September 26, 2014 at 11:24 AM
To: Andrew Davidson <An...@SantaCruzIntegration.com>
Cc: Davies Liu <da...@databricks.com>, "user@spark.apache.org"
<us...@spark.apache.org>
Subject: Re: problem with spark-ec2 launch script Re: spark-ec2 ERROR: Line
magic function `%matplotlib` not found
> Are you able to use the regular PySpark shell on your EC2 cluster? That would
> be the first thing to confirm is working.
>
> I don’t know whether the version of Python on the cluster would affect whether
> IPython works or not, but if you want to try manually upgrading Python on a
> cluster launched by spark-ec2, there are some instructions in the comments
> here <https://issues.apache.org/jira/browse/SPARK-922> for doing so.
>
> Nick
>
>
>
> On Fri, Sep 26, 2014 at 2:18 PM, Andy Davidson <An...@santacruzintegration.com>
> wrote:
>> Hi Davies
>>
>> The real issue is about cluster management. I am new to the spark world and
>> am not a system administrator. It seem like the problem is with the
>> spark-ec2 launch script. It is installing old version of python
>>
>> In the mean time I am trying to figure out how I can manually install the
>> correct version on all the machines in my cluster
>>
>> Thanks
>>
>> Andy
>>
>> From: Davies Liu <da...@databricks.com>
>> Date: Thursday, September 25, 2014 at 9:58 PM
>> To: Andrew Davidson <An...@SantaCruzIntegration.com>
>> Cc: "user@spark.apache.org" <us...@spark.apache.org>
>> Subject: Re: spark-ec2 ERROR: Line magic function `%matplotlib` not found
>>
>>> Maybe you have Python 2.7 on master but Python 2.6 in cluster,
>>> you should upgrade python to 2.7 in cluster, or use python 2.6 in
>>> master by set PYSPARK_PYTHON=python2.6
>>>
>>> On Thu, Sep 25, 2014 at 5:11 PM, Andy Davidson
>>> <An...@santacruzintegration.com> wrote:
>>>> Hi
>>>>
>>>> I am running into trouble using iPython notebook on my cluster. Use the
>>>> following command to set the cluster up
>>>>
>>>> $ ./spark-ec2 --key-pair=$KEY_PAIR --identity-file=$KEY_FILE
>>>> --region=$REGION --slaves=$NUM_SLAVES launch $CLUSTER_NAME
>>>>
>>>>
>>>> On master I launch python as follows
>>>>
>>>> $ IPYTHON_OPTS="notebook --pylab inline --no-browser --port=7000"
>>>> $SPARK_HOME/bin/pyspark
>>>>
>>>>
>>>> It looks like the problem is the cluster is using an old version of python
>>>> and python. Any idea how I can easily upgrade ? The following version
>>>> works
>>>> on my mac
>>>>
>>>> Thanks
>>>>
>>>> Andy
>>>>
>>>> {'commit_hash': '681fd77',
>>>> 'commit_source': 'installation',
>>>> 'default_encoding': 'UTF-8',
>>>> 'ipython_path': '/Library/Python/2.7/site-packages/IPython',
>>>> 'ipython_version': '2.1.0',
>>>> 'os_name': 'posix',
>>>> 'platform': 'Darwin-13.3.0-x86_64-i386-64bit',
>>>> 'sys_executable': '/usr/bin/python',
>>>> 'sys_platform': 'darwin',
>>>> 'sys_version': '2.7.5 (default, Mar 9 2014, 22:15:05) \n[GCC 4.2.1
>>>> Compatible Apple LLVM 5.0 (clang-500.0.68)]’}
>>>>
>>>>
>>>>
>>>>
>>>
>
Re: problem with spark-ec2 launch script Re: spark-ec2 ERROR: Line
magic function `%matplotlib` not found
Posted by Nicholas Chammas <ni...@gmail.com>.
Are you able to use the regular PySpark shell on your EC2 cluster? That
would be the first thing to confirm is working.
I don’t know whether the version of Python on the cluster would affect
whether IPython works or not, but if you want to try manually upgrading
Python on a cluster launched by spark-ec2, there are some instructions in
the comments here <https://issues.apache.org/jira/browse/SPARK-922> for
doing so.
Nick
On Fri, Sep 26, 2014 at 2:18 PM, Andy Davidson <
Andy@santacruzintegration.com> wrote:
> Hi Davies
>
> The real issue is about cluster management. I am new to the spark world
> and am not a system administrator. It seem like the problem is with the
> spark-ec2 launch script. It is installing old version of python
>
> In the mean time I am trying to figure out how I can manually install the
> correct version on all the machines in my cluster
>
> Thanks
>
> Andy
>
> From: Davies Liu <da...@databricks.com>
> Date: Thursday, September 25, 2014 at 9:58 PM
> To: Andrew Davidson <An...@SantaCruzIntegration.com>
> Cc: "user@spark.apache.org" <us...@spark.apache.org>
> Subject: Re: spark-ec2 ERROR: Line magic function `%matplotlib` not found
>
> Maybe you have Python 2.7 on master but Python 2.6 in cluster,
> you should upgrade python to 2.7 in cluster, or use python 2.6 in
> master by set PYSPARK_PYTHON=python2.6
>
> On Thu, Sep 25, 2014 at 5:11 PM, Andy Davidson
> <An...@santacruzintegration.com> wrote:
>
> Hi
>
> I am running into trouble using iPython notebook on my cluster. Use the
> following command to set the cluster up
>
> $ ./spark-ec2 --key-pair=$KEY_PAIR --identity-file=$KEY_FILE
> --region=$REGION --slaves=$NUM_SLAVES launch $CLUSTER_NAME
>
>
> On master I launch python as follows
>
> $ IPYTHON_OPTS="notebook --pylab inline --no-browser --port=7000"
> $SPARK_HOME/bin/pyspark
>
>
> It looks like the problem is the cluster is using an old version of python
> and python. Any idea how I can easily upgrade ? The following version works
> on my mac
>
> Thanks
>
> Andy
>
> {'commit_hash': '681fd77',
> 'commit_source': 'installation',
> 'default_encoding': 'UTF-8',
> 'ipython_path': '/Library/Python/2.7/site-packages/IPython',
> 'ipython_version': '2.1.0',
> 'os_name': 'posix',
> 'platform': 'Darwin-13.3.0-x86_64-i386-64bit',
> 'sys_executable': '/usr/bin/python',
> 'sys_platform': 'darwin',
> 'sys_version': '2.7.5 (default, Mar 9 2014, 22:15:05) \n[GCC 4.2.1
> Compatible Apple LLVM 5.0 (clang-500.0.68)]’}
>
>
>
>
>
>