You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Suvayu Ali <fa...@gmail.com> on 2017/07/02 12:01:44 UTC

Installing PyArrow on Amazon Linux

Hi Arrow devs,

I'm not sure if this is the correct place to ask, if not, please point
me in the right direction.

I wanted to use the HDFS client with PySpark (for now).  My Spark
cluster is on Amazon EMR, so the nodes use Amazon Linux (2017.03).  On
my dev machine (Fedora 25) with Python 3.5.1, a simple `pip3 install
pyarrow` works just fine, and I can use the HDFS client.  But when I do
the same on Amazon Linux, it complains about incompatible Python
version, then pip tries to download an old tarball (0.3), and finally
cmake fails with missing cmake files (PythonLibsNewConfig.cmake).

Here's a snippet from the failure with verbose on:

  $ sudo python3.5 -m pip -vvv install pyarrow
  [..]
    Skipping https://pypi.python.org/../pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl#md5=.. 
    (from https://pypi.python.org/simple/pyarrow/) because it is not compatible with this Python
    Skipping https://pypi.python.org/../pyarrow-0.4.0-cp27-cp27mu-manylinux1_x86_64.whl#md5=..
    (from https://pypi.python.org/simple/pyarrow/) because it is not compatible with this Python
    Skipping https://pypi.python.org/../pyarrow-0.2.0-cp34-cp34m-manylinux1_x86_64.whl#md5=..
    (from https://pypi.python.org/simple/pyarrow/) because it is not compatible with this Python
    Using version 0.3.0 (newest of versions: 0.3.0, 0.2.0)
  [..]

When I explicitly specify which wheel file to use, it fails like this:

  $ sudo python3.5 -m pip -vvv install https://pypi.python.org/../pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl#md5=..
  pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl is not a supported wheel on this platform.
  Exception information:
  Traceback (most recent call last):
    File "/usr/lib/python3.5/dist-packages/pip/basecommand.py", line 246, in main
      status = self.run(options, args)
    File "/usr/lib/python3.5/dist-packages/pip/commands/install.py", line 308, in run
      name, None, isolated=options.isolated_mode,
    File "/usr/lib/python3.5/dist-packages/pip/req/req_install.py", line 207, in from_line
      wheel.filename
  pip.exceptions.UnsupportedWheel: pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl is not a supported wheel on this platform.

Any ideas what is going wrong?  Thanks for any help.

Cheers,

-- 
Suvayu

Open source is the future. It sets us free.

Re: Installing PyArrow on Amazon Linux

Posted by Suvayu Ali <fa...@gmail.com>.
Hello Uwe,

On Sun, Jul 02, 2017 at 02:15:38PM +0200, Uwe L. Korn wrote:
> 
> 1. Your pip is too old, you need at least 8.1.2

That was it :).  Thanks a lot!

Cheers,

-- 
Suvayu

Open source is the future. It sets us free.

Re: Installing PyArrow on Amazon Linux

Posted by "Uwe L. Korn" <uw...@xhochy.com>.
Hello,

there are two possibilities that directly come to my mind:

1. Your pip is too old, you need at least 8.1.2
2. You're on a 32bit system

Given that https://aws.amazon.com/amazon-linux-ami/2017.03-packages/
states python-pip-6.1.1, mostly likely option 1 is your problem and can
be solved by upgrading to the latest pip version.

Uwe

On Sun, Jul 2, 2017, at 02:01 PM, Suvayu Ali wrote:
> Hi Arrow devs,
> 
> I'm not sure if this is the correct place to ask, if not, please point
> me in the right direction.
> 
> I wanted to use the HDFS client with PySpark (for now).  My Spark
> cluster is on Amazon EMR, so the nodes use Amazon Linux (2017.03).  On
> my dev machine (Fedora 25) with Python 3.5.1, a simple `pip3 install
> pyarrow` works just fine, and I can use the HDFS client.  But when I do
> the same on Amazon Linux, it complains about incompatible Python
> version, then pip tries to download an old tarball (0.3), and finally
> cmake fails with missing cmake files (PythonLibsNewConfig.cmake).
> 
> Here's a snippet from the failure with verbose on:
> 
>   $ sudo python3.5 -m pip -vvv install pyarrow
>   [..]
>     Skipping
>     https://pypi.python.org/../pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl#md5=.. 
>     (from https://pypi.python.org/simple/pyarrow/) because it is not
>     compatible with this Python
>     Skipping
>     https://pypi.python.org/../pyarrow-0.4.0-cp27-cp27mu-manylinux1_x86_64.whl#md5=..
>     (from https://pypi.python.org/simple/pyarrow/) because it is not
>     compatible with this Python
>     Skipping
>     https://pypi.python.org/../pyarrow-0.2.0-cp34-cp34m-manylinux1_x86_64.whl#md5=..
>     (from https://pypi.python.org/simple/pyarrow/) because it is not
>     compatible with this Python
>     Using version 0.3.0 (newest of versions: 0.3.0, 0.2.0)
>   [..]
> 
> When I explicitly specify which wheel file to use, it fails like this:
> 
>   $ sudo python3.5 -m pip -vvv install
>   https://pypi.python.org/../pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl#md5=..
>   pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl is not a supported wheel
>   on this platform.
>   Exception information:
>   Traceback (most recent call last):
>     File "/usr/lib/python3.5/dist-packages/pip/basecommand.py", line 246,
>     in main
>       status = self.run(options, args)
>     File "/usr/lib/python3.5/dist-packages/pip/commands/install.py", line
>     308, in run
>       name, None, isolated=options.isolated_mode,
>     File "/usr/lib/python3.5/dist-packages/pip/req/req_install.py", line
>     207, in from_line
>       wheel.filename
>   pip.exceptions.UnsupportedWheel:
>   pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl is not a supported wheel
>   on this platform.
> 
> Any ideas what is going wrong?  Thanks for any help.
> 
> Cheers,
> 
> -- 
> Suvayu
> 
> Open source is the future. It sets us free.