You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "nicholas.chammas" <ni...@gmail.com> on 2014/02/28 00:48:42 UTC
Running Spark with Python 2.7.5+
The provided Spark EC2
scripts<https://spark.incubator.apache.org/docs/0.9.0/ec2-scripts.html>and
default AMI ship with Python 2.6.8.
I would like to use Python 2.7.5 or later. I believe that among the 2.x
versions, 2.7 is the most popular.
What's the easiest way to get my Spark cluster on Python 2.7 and working
correctly?
For starters, I naively upgraded Python on the master node using the
instructions here<http://www.lecloud.net/post/61401763496/install-update-to-python-2-7-and-latest-pip-on-ec2>.
The pyspark shell started up fine but a basic map on an RDD that used to
work fine failed. Perhaps I just need to upgrade all the nodes in my
cluster, and perhaps it's not that simple. I thought I'd check here first.
Nick
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-with-Python-2-7-5-tp2131.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Running Spark with Python 2.7.5+
Posted by Nicholas Chammas <ni...@gmail.com>.
Perfect. Thanks Josh. I've added myself as a watcher on the ticket.
(By the way, when I upgraded to 2.7 I replaced 2.6 so the executable name
didn't change.)
On Fri, Feb 28, 2014 at 12:12 AM, Josh Rosen <ro...@gmail.com> wrote:
> There's an open ticket to update the Python version:
> https://spark-project.atlassian.net/browse/SPARK-922. In that ticket, I
> included instructions for a workaround to manually update a cluster to
> Python 2.7.
>
> Did you set the PYSPARK_PYTHON environment variable to the name of your
> new Python executable? You may have to do this if Python 2.7 executable
> was installed as "python27" instead of "python."
>
>
> On Thu, Feb 27, 2014 at 9:03 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> Makes sense. I'll give it a shot and check back here if that doesn't work.
>>
>> Are there plans to upgrade the EC2 deployment scripts and/or AMI to have
>> Python 2.7 by default? If so, is there a ticket somewhere I can follow?
>>
>> Nick
>>
>>
>> On Thu, Feb 27, 2014 at 6:50 PM, Bryn Keller <xo...@xoltar.org> wrote:
>>
>>> Hi Nick,
>>>
>>> All the nodes of the cluster need to have the same Python setup (path
>>> and version). So if, e.g. you start running in 2.7.5 on the master and it
>>> ships code to nodes that have 2.6.x, you'll get invalid opcode errors.
>>>
>>> Thanks,
>>> Bryn
>>>
>>>
>>> On Thu, Feb 27, 2014 at 3:48 PM, nicholas.chammas <
>>> nicholas.chammas@gmail.com> wrote:
>>>
>>>> The provided Spark EC2 scripts<https://spark.incubator.apache.org/docs/0.9.0/ec2-scripts.html>and default AMI ship with Python 2.6.8.
>>>>
>>>> I would like to use Python 2.7.5 or later. I believe that among the 2.x
>>>> versions, 2.7 is the most popular.
>>>>
>>>> What's the easiest way to get my Spark cluster on Python 2.7 and
>>>> working correctly?
>>>>
>>>> For starters, I naively upgraded Python on the master node using the
>>>> instructions here<http://www.lecloud.net/post/61401763496/install-update-to-python-2-7-and-latest-pip-on-ec2>.
>>>> The pyspark shell started up fine but a basic map on an RDD that used to
>>>> work fine failed. Perhaps I just need to upgrade all the nodes in my
>>>> cluster, and perhaps it's not that simple. I thought I'd check here first.
>>>>
>>>> Nick
>>>>
>>>>
>>>> ------------------------------
>>>> View this message in context: Running Spark with Python 2.7.5+<http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-with-Python-2-7-5-tp2131.html>
>>>> Sent from the Apache Spark User List mailing list archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at Nabble.com.
>>>>
>>>
>>>
>>
>
Re: Running Spark with Python 2.7.5+
Posted by Josh Rosen <ro...@gmail.com>.
There's an open ticket to update the Python version:
https://spark-project.atlassian.net/browse/SPARK-922. In that ticket, I
included instructions for a workaround to manually update a cluster to
Python 2.7.
Did you set the PYSPARK_PYTHON environment variable to the name of your new
Python executable? You may have to do this if Python 2.7 executable was
installed as "python27" instead of "python."
On Thu, Feb 27, 2014 at 9:03 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> Makes sense. I'll give it a shot and check back here if that doesn't work.
>
> Are there plans to upgrade the EC2 deployment scripts and/or AMI to have
> Python 2.7 by default? If so, is there a ticket somewhere I can follow?
>
> Nick
>
>
> On Thu, Feb 27, 2014 at 6:50 PM, Bryn Keller <xo...@xoltar.org> wrote:
>
>> Hi Nick,
>>
>> All the nodes of the cluster need to have the same Python setup (path and
>> version). So if, e.g. you start running in 2.7.5 on the master and it ships
>> code to nodes that have 2.6.x, you'll get invalid opcode errors.
>>
>> Thanks,
>> Bryn
>>
>>
>> On Thu, Feb 27, 2014 at 3:48 PM, nicholas.chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> The provided Spark EC2 scripts<https://spark.incubator.apache.org/docs/0.9.0/ec2-scripts.html>and default AMI ship with Python 2.6.8.
>>>
>>> I would like to use Python 2.7.5 or later. I believe that among the 2.x
>>> versions, 2.7 is the most popular.
>>>
>>> What's the easiest way to get my Spark cluster on Python 2.7 and working
>>> correctly?
>>>
>>> For starters, I naively upgraded Python on the master node using the
>>> instructions here<http://www.lecloud.net/post/61401763496/install-update-to-python-2-7-and-latest-pip-on-ec2>.
>>> The pyspark shell started up fine but a basic map on an RDD that used to
>>> work fine failed. Perhaps I just need to upgrade all the nodes in my
>>> cluster, and perhaps it's not that simple. I thought I'd check here first.
>>>
>>> Nick
>>>
>>>
>>> ------------------------------
>>> View this message in context: Running Spark with Python 2.7.5+<http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-with-Python-2-7-5-tp2131.html>
>>> Sent from the Apache Spark User List mailing list archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at Nabble.com.
>>>
>>
>>
>
Re: Running Spark with Python 2.7.5+
Posted by Nicholas Chammas <ni...@gmail.com>.
Makes sense. I'll give it a shot and check back here if that doesn't work.
Are there plans to upgrade the EC2 deployment scripts and/or AMI to have
Python 2.7 by default? If so, is there a ticket somewhere I can follow?
Nick
On Thu, Feb 27, 2014 at 6:50 PM, Bryn Keller <xo...@xoltar.org> wrote:
> Hi Nick,
>
> All the nodes of the cluster need to have the same Python setup (path and
> version). So if, e.g. you start running in 2.7.5 on the master and it ships
> code to nodes that have 2.6.x, you'll get invalid opcode errors.
>
> Thanks,
> Bryn
>
>
> On Thu, Feb 27, 2014 at 3:48 PM, nicholas.chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> The provided Spark EC2 scripts<https://spark.incubator.apache.org/docs/0.9.0/ec2-scripts.html>and default AMI ship with Python 2.6.8.
>>
>> I would like to use Python 2.7.5 or later. I believe that among the 2.x
>> versions, 2.7 is the most popular.
>>
>> What's the easiest way to get my Spark cluster on Python 2.7 and working
>> correctly?
>>
>> For starters, I naively upgraded Python on the master node using the
>> instructions here<http://www.lecloud.net/post/61401763496/install-update-to-python-2-7-and-latest-pip-on-ec2>.
>> The pyspark shell started up fine but a basic map on an RDD that used to
>> work fine failed. Perhaps I just need to upgrade all the nodes in my
>> cluster, and perhaps it's not that simple. I thought I'd check here first.
>>
>> Nick
>>
>>
>> ------------------------------
>> View this message in context: Running Spark with Python 2.7.5+<http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-with-Python-2-7-5-tp2131.html>
>> Sent from the Apache Spark User List mailing list archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at Nabble.com.
>>
>
>
Re: Running Spark with Python 2.7.5+
Posted by Bryn Keller <xo...@xoltar.org>.
Hi Nick,
All the nodes of the cluster need to have the same Python setup (path and
version). So if, e.g. you start running in 2.7.5 on the master and it ships
code to nodes that have 2.6.x, you'll get invalid opcode errors.
Thanks,
Bryn
On Thu, Feb 27, 2014 at 3:48 PM, nicholas.chammas <
nicholas.chammas@gmail.com> wrote:
> The provided Spark EC2 scripts<https://spark.incubator.apache.org/docs/0.9.0/ec2-scripts.html>and default AMI ship with Python 2.6.8.
>
> I would like to use Python 2.7.5 or later. I believe that among the 2.x
> versions, 2.7 is the most popular.
>
> What's the easiest way to get my Spark cluster on Python 2.7 and working
> correctly?
>
> For starters, I naively upgraded Python on the master node using the
> instructions here<http://www.lecloud.net/post/61401763496/install-update-to-python-2-7-and-latest-pip-on-ec2>.
> The pyspark shell started up fine but a basic map on an RDD that used to
> work fine failed. Perhaps I just need to upgrade all the nodes in my
> cluster, and perhaps it's not that simple. I thought I'd check here first.
>
> Nick
>
>
> ------------------------------
> View this message in context: Running Spark with Python 2.7.5+<http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-with-Python-2-7-5-tp2131.html>
> Sent from the Apache Spark User List mailing list archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at Nabble.com.
>