You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by Xander Dunn <xa...@xander.ai> on 2021/04/13 19:59:30 UTC
[Python] pyarrow.gandiva unavailable on Ubuntu?
On my local macOS 11.2.3:
```
$ python --version
Python 3.7.10
$ pip --version
pip 21.0.1 from /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pip (python 3.7)
$ pip list | grep pyarrow
pyarrow 3.0.0
$ which python
/usr/local/anaconda3/envs/my_model/bin/python
$ python
>>> import pyarrow as pa
>>> print(pa.__file__)
/ usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py ( http://usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py )
>>> import pyarrow.plasma
>>> import pyarrow.gandiva as ga
>>> print(ga.__file__)
/ usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/gandiva.cpython-37m-darwin.so ( http://usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/gandiva.cpython-37m-darwin.so )
```
On my Ubuntu 14.04 instance:
```
$ python --version
Python 3.7.10
$ pip --version
pip 21.0.1 from /home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pip (python 3.7)
$ pip list | grep pyarrow
pyarrow 3.0.0
$ which python
/home/xander/anaconda3/envs/my_model/bin/python
$ python
>>> import pyarrow as pa
>>> print(pa.__file__)
/ home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py ( http://home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py )
>>> import pyarrow.plasma
>>> import pyarrow.gondiva
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pyarrow.gondiva'
```
You can see that pyarrow.gondiva is found on mac but not on Ubuntu. Same Python version. Same pyarrow version. I installed both of them with `conda install -c conda-forge pyarrow==3.0.0`.
On Mac, I see the expected Cython file and library:
```
$ l /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ | grep gandiva
-rwxrwxr-x 2 xander staff 221K Apr 1 12:44 gandiva.cpython-37m-darwin.so ( http://gandiva.cpython-37m-darwin.so/ )
-rw-rw-r-- 2 xander staff 17K Jan 18 14:00 gandiva.pyx
```
On Ubuntu, I see only the Cython file:
```
$ l ~/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ | grep gandiva
-rw-rw-r-- 1 xander xander 17K Apr 13 12:28 gandiva.pyx
```
Is this expected? Should I be able to import pyarrow.gandiva on Ubuntu? Everything is run on Ubuntu so if I make use of pyarrow.gandiva I'll need to figure out how to call it.
It's mentioned here that it was removed from Python wheels but should still be available in the conda install: https://issues.apache.org/jira/browse/ARROW-10154. I'm not finding it in my Ubuntu conda install.
Thanks,
Xander
Re: [Python] pyarrow.gandiva unavailable on Ubuntu?
Posted by Xander Dunn <xa...@xander.ai>.
One odd thing I noticed is that on Mac, calling ` conda install -c conda-forge pyarrow ` defaults to installing pyarrow 3.0.0. However, on Ubuntu it defaults to installing pyarrow 0.11.1, which is why above I've been specifying '==3.0.0` on all of the conda installs. I don't know why conda on Ubuntu is attempting to install pyarrow 0.11.1 rather than latest.
On Tue, Apr 13, 2021 at 1:45 PM, Xander Dunn < xander@xander.ai > wrote:
>
> ```
>
> $ conda create -n fresh_env python=3.7
>
> $ conda activate fresh_env
>
> $ conda install -c conda-forge pyarrow==3.0.0
>
> $ python
>
> >>> import pyarrow as pa
>
> >>> pa.__version__
>
> '3.0.0'
>
> >>> pa.__file__
>
> '/ home/ xander/ anaconda3/ envs/ fresh_env/ lib/ python3. 7/ site-packages/
> pyarrow/ __init__. py (
> http://home/xander/anaconda3/envs/fresh_env/lib/python3.7/site-packages/pyarrow/__init__.py
> ) '
>
> >>> import pyarrow.plasma
>
> >>> import pyarrow.gandiva
>
> Traceback (most recent call last):
>
> File "<stdin>", line 1, in <module>
>
> ModuleNotFoundError: No module named 'pyarrow.gandiva'
>
> ```
>
> Same result in a fresh conda environment. The library is also still
> missing from the installed location:
>
> ```
>
> $ l
> /home/xander/anaconda3/envs/fresh_env/lib/python3.7/site-packages/pyarrow/
> | grep gandiva
>
> -rw-rw-r-- 3 xander xander 17K Jan 18 14:00 gandiva.pyx
>
> ```
>
>
>
> And a correction to what I wrote earlier, I'm on Ubuntu 18.04:
>
> ```
>
> $ lsb_release -a
>
> No LSB modules are available.
>
> Distributor ID: Ubuntu
>
> Description: Ubuntu 18.04.5 LTS
>
> Release: 18.04
>
> Codename: bionic
>
> $ conda --version
>
> conda 4.10.0
>
> ```
>
>
>
> I do have several arrow packages installed via apt, but I don't think
> that's interfering:
>
> ```
>
> apache-arrow-archive-keyring/bionic,now 3.0.0-1 all [installed]
>
> gir1.2-arrow-1.0/bionic,now 3.0.0-1 amd64 [installed,automatic]
>
> libarrow-dev/bionic,now 3.0.0-1 amd64 [installed]
>
> libarrow-glib-dev/bionic,now 3.0.0-1 amd64 [installed,automatic]
>
> libarrow-glib300/bionic,now 3.0.0-1 amd64 [installed,automatic]
>
> libarrow300/bionic,now 3.0.0-1 amd64 [installed]
>
> ```
>
>
>
> Thanks for the help,
>
> Xander
>
>
>
>
>
> On Tue, Apr 13, 2021 at 1:25 PM, Wes McKinney < wesmckinn@ gmail. com (
> wesmckinn@gmail.com ) > wrote:
>
>> It looks to me like you have the wheel installed, not the conda package.
>> Can you reproduce this on Ubuntu from a fresh conda environment?
>>
>> On Tue, Apr 13, 2021 at 3:16 PM Xander Dunn < xander@ xander. ai (
>> xander@xander.ai ) > wrote:
>>
>>
>>> Typo. The issue remains present. From my Ubuntu machine just now:
>>>
>>> ```
>>>
>>> $ python
>>>
>>> >>> import pyarrow as pa
>>>
>>> >>> print(pa.__file__)
>>>
>>> / home/ xander/ anaconda3/ envs/ plutus_model/ lib/ python3. 7/ site-packages/
>>> pyarrow/ __init__. py (
>>> http://home/xander/anaconda3/envs/plutus_model/lib/python3.7/site-packages/pyarrow/__init__.py
>>> )
>>>
>>> >>> import pyarrow.plasma
>>>
>>> >>> import pyarrow.gandiva
>>>
>>> Traceback (most recent call last):
>>>
>>> File "<stdin>", line 1, in <module>
>>>
>>> ModuleNotFoundError: No module named 'pyarrow.gandiva'
>>>
>>> ```
>>>
>>>
>>>
>>> The .py I'm executing on both machines is identical. Works on mac. Not
>>> found on Ubuntu.
>>>
>>>
>>>
>>>
>>> On Tue, Apr 13, 2021 at 1:01 PM, Micah Kornfield < emkornfield@ gmail. com
>>> ( emkornfield@gmail.com ) > wrote:
>>>
>>>> Hi Xander,
>>>> Was there autocorrect on this e-mail? the second example shows "gondiva"
>>>> not "gandiva"
>>>>
>>>>
>>>> -Micah
>>>>
>>>> On Tue, Apr 13, 2021 at 12:59 PM Xander Dunn < xander@ xander. ai (
>>>> xander@xander.ai ) > wrote:
>>>>
>>>>
>>>>> On my local macOS 11.2.3:
>>>>>
>>>>> ```
>>>>>
>>>>> $ python --version
>>>>>
>>>>> Python 3.7.10
>>>>>
>>>>> $ pip --version
>>>>>
>>>>> pip 21.0.1 from
>>>>> /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pip (python
>>>>> 3.7)
>>>>>
>>>>> $ pip list | grep pyarrow
>>>>>
>>>>> pyarrow 3.0.0
>>>>>
>>>>> $ which python
>>>>>
>>>>> /usr/local/anaconda3/envs/my_model/bin/python
>>>>>
>>>>> $ python
>>>>>
>>>>> >>> import pyarrow as pa
>>>>>
>>>>> >>> print(pa.__file__)
>>>>>
>>>>> / usr/ local/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/ pyarrow/
>>>>> __init__. py (
>>>>> http://usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py
>>>>> )
>>>>>
>>>>> >>> import pyarrow.plasma
>>>>>
>>>>> >>> import pyarrow.gandiva as ga
>>>>>
>>>>> >>> print(ga.__file__)
>>>>>
>>>>> / usr/ local/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/ pyarrow/
>>>>> gandiva. cpython-37m-darwin. so (
>>>>> http://usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/gandiva.cpython-37m-darwin.so
>>>>> )
>>>>>
>>>>> ```
>>>>>
>>>>>
>>>>>
>>>>> On my Ubuntu 14.04 instance:
>>>>>
>>>>> ```
>>>>>
>>>>> $ python --version
>>>>>
>>>>> Python 3.7.10
>>>>>
>>>>> $ pip --version
>>>>>
>>>>> pip 21.0.1 from
>>>>> /home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pip
>>>>> (python 3.7)
>>>>>
>>>>> $ pip list | grep pyarrow
>>>>>
>>>>> pyarrow 3.0.0
>>>>>
>>>>> $ which python
>>>>>
>>>>> /home/xander/anaconda3/envs/my_model/bin/python
>>>>>
>>>>> $ python
>>>>>
>>>>> >>> import pyarrow as pa
>>>>>
>>>>> >>> print(pa.__file__)
>>>>>
>>>>> / home/ xander/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/
>>>>> pyarrow/ __init__. py (
>>>>> http://home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py
>>>>> )
>>>>>
>>>>> >>> import pyarrow.plasma
>>>>>
>>>>> >>> import pyarrow.gondiva
>>>>>
>>>>> Traceback (most recent call last):
>>>>>
>>>>> File "<stdin>", line 1, in <module>
>>>>>
>>>>> ModuleNotFoundError: No module named 'pyarrow.gondiva'
>>>>>
>>>>> ```
>>>>>
>>>>> You can see that pyarrow.gondiva is found on mac but not on Ubuntu. Same
>>>>> Python version. Same pyarrow version. I installed both of them with `conda
>>>>> install -c conda-forge pyarrow==3.0.0`.
>>>>>
>>>>>
>>>>>
>>>>> On Mac, I see the expected Cython file and library:
>>>>>
>>>>> ```
>>>>>
>>>>> $ l
>>>>> /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ |
>>>>> grep gandiva
>>>>>
>>>>> -rwxrwxr-x 2 xander staff 221K Apr 1 12:44 gandiva. cpython-37m-darwin.
>>>>> so ( http://gandiva.cpython-37m-darwin.so/ )
>>>>>
>>>>> -rw-rw-r-- 2 xander staff 17K Jan 18 14:00 gandiva.pyx
>>>>>
>>>>> ```
>>>>>
>>>>>
>>>>>
>>>>> On Ubuntu, I see only the Cython file:
>>>>>
>>>>> ```
>>>>>
>>>>> $ l ~/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ | grep
>>>>> gandiva
>>>>>
>>>>> -rw-rw-r-- 1 xander xander 17K Apr 13 12:28 gandiva.pyx
>>>>>
>>>>> ```
>>>>>
>>>>>
>>>>>
>>>>> Is this expected? Should I be able to import pyarrow.gandiva on Ubuntu?
>>>>> Everything is run on Ubuntu so if I make use of pyarrow.gandiva I'll need
>>>>> to figure out how to call it.
>>>>>
>>>>>
>>>>>
>>>>> It's mentioned here that it was removed from Python wheels but should
>>>>> still be available in the conda install: https:/ / issues. apache. org/ jira/
>>>>> browse/ ARROW-10154 ( https://issues.apache.org/jira/browse/ARROW-10154 ).
>>>>> I'm not finding it in my Ubuntu conda install.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Xander
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: [Python] pyarrow.gandiva unavailable on Ubuntu?
Posted by Xander Dunn <xa...@xander.ai>.
```
$ conda create -n fresh_env python=3.7
$ conda activate fresh_env
$ conda install -c conda-forge pyarrow==3.0.0
$ python
>>> import pyarrow as pa
>>> pa.__version__
'3.0.0'
>>> pa.__file__
'/ home/xander/anaconda3/envs/fresh_env/lib/python3.7/site-packages/pyarrow/__init__.py ( http://home/xander/anaconda3/envs/fresh_env/lib/python3.7/site-packages/pyarrow/__init__.py ) '
>>> import pyarrow.plasma
>>> import pyarrow.gandiva
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pyarrow.gandiva'
```
Same result in a fresh conda environment. The library is also still missing from the installed location:
```
$ l /home/xander/anaconda3/envs/fresh_env/lib/python3.7/site-packages/pyarrow/ | grep gandiva
-rw-rw-r-- 3 xander xander 17K Jan 18 14:00 gandiva.pyx
```
And a correction to what I wrote earlier, I'm on Ubuntu 18.04:
```
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic
$ conda --version
conda 4.10.0
```
I do have several arrow packages installed via apt, but I don't think that's interfering:
```
apache-arrow-archive-keyring/bionic,now 3.0.0-1 all [installed]
gir1.2-arrow-1.0/bionic,now 3.0.0-1 amd64 [installed,automatic]
libarrow-dev/bionic,now 3.0.0-1 amd64 [installed]
libarrow-glib-dev/bionic,now 3.0.0-1 amd64 [installed,automatic]
libarrow-glib300/bionic,now 3.0.0-1 amd64 [installed,automatic]
libarrow300/bionic,now 3.0.0-1 amd64 [installed]
```
Thanks for the help,
Xander
On Tue, Apr 13, 2021 at 1:25 PM, Wes McKinney < wesmckinn@gmail.com > wrote:
>
> It looks to me like you have the wheel installed, not the conda package.
> Can you reproduce this on Ubuntu from a fresh conda environment?
>
> On Tue, Apr 13, 2021 at 3:16 PM Xander Dunn < xander@ xander. ai (
> xander@xander.ai ) > wrote:
>
>
>> Typo. The issue remains present. From my Ubuntu machine just now:
>>
>> ```
>>
>> $ python
>>
>> >>> import pyarrow as pa
>>
>> >>> print(pa.__file__)
>>
>> / home/ xander/ anaconda3/ envs/ plutus_model/ lib/ python3. 7/ site-packages/
>> pyarrow/ __init__. py (
>> http://home/xander/anaconda3/envs/plutus_model/lib/python3.7/site-packages/pyarrow/__init__.py
>> )
>>
>> >>> import pyarrow.plasma
>>
>> >>> import pyarrow.gandiva
>>
>> Traceback (most recent call last):
>>
>> File "<stdin>", line 1, in <module>
>>
>> ModuleNotFoundError: No module named 'pyarrow.gandiva'
>>
>> ```
>>
>>
>>
>> The .py I'm executing on both machines is identical. Works on mac. Not
>> found on Ubuntu.
>>
>>
>>
>>
>> On Tue, Apr 13, 2021 at 1:01 PM, Micah Kornfield < emkornfield@ gmail. com
>> ( emkornfield@gmail.com ) > wrote:
>>
>>> Hi Xander,
>>> Was there autocorrect on this e-mail? the second example shows "gondiva"
>>> not "gandiva"
>>>
>>>
>>> -Micah
>>>
>>> On Tue, Apr 13, 2021 at 12:59 PM Xander Dunn < xander@ xander. ai (
>>> xander@xander.ai ) > wrote:
>>>
>>>
>>>> On my local macOS 11.2.3:
>>>>
>>>> ```
>>>>
>>>> $ python --version
>>>>
>>>> Python 3.7.10
>>>>
>>>> $ pip --version
>>>>
>>>> pip 21.0.1 from
>>>> /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pip (python
>>>> 3.7)
>>>>
>>>> $ pip list | grep pyarrow
>>>>
>>>> pyarrow 3.0.0
>>>>
>>>> $ which python
>>>>
>>>> /usr/local/anaconda3/envs/my_model/bin/python
>>>>
>>>> $ python
>>>>
>>>> >>> import pyarrow as pa
>>>>
>>>> >>> print(pa.__file__)
>>>>
>>>> / usr/ local/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/ pyarrow/
>>>> __init__. py (
>>>> http://usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py
>>>> )
>>>>
>>>> >>> import pyarrow.plasma
>>>>
>>>> >>> import pyarrow.gandiva as ga
>>>>
>>>> >>> print(ga.__file__)
>>>>
>>>> / usr/ local/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/ pyarrow/
>>>> gandiva. cpython-37m-darwin. so (
>>>> http://usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/gandiva.cpython-37m-darwin.so
>>>> )
>>>>
>>>> ```
>>>>
>>>>
>>>>
>>>> On my Ubuntu 14.04 instance:
>>>>
>>>> ```
>>>>
>>>> $ python --version
>>>>
>>>> Python 3.7.10
>>>>
>>>> $ pip --version
>>>>
>>>> pip 21.0.1 from
>>>> /home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pip
>>>> (python 3.7)
>>>>
>>>> $ pip list | grep pyarrow
>>>>
>>>> pyarrow 3.0.0
>>>>
>>>> $ which python
>>>>
>>>> /home/xander/anaconda3/envs/my_model/bin/python
>>>>
>>>> $ python
>>>>
>>>> >>> import pyarrow as pa
>>>>
>>>> >>> print(pa.__file__)
>>>>
>>>> / home/ xander/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/
>>>> pyarrow/ __init__. py (
>>>> http://home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py
>>>> )
>>>>
>>>> >>> import pyarrow.plasma
>>>>
>>>> >>> import pyarrow.gondiva
>>>>
>>>> Traceback (most recent call last):
>>>>
>>>> File "<stdin>", line 1, in <module>
>>>>
>>>> ModuleNotFoundError: No module named 'pyarrow.gondiva'
>>>>
>>>> ```
>>>>
>>>> You can see that pyarrow.gondiva is found on mac but not on Ubuntu. Same
>>>> Python version. Same pyarrow version. I installed both of them with `conda
>>>> install -c conda-forge pyarrow==3.0.0`.
>>>>
>>>>
>>>>
>>>> On Mac, I see the expected Cython file and library:
>>>>
>>>> ```
>>>>
>>>> $ l
>>>> /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ |
>>>> grep gandiva
>>>>
>>>> -rwxrwxr-x 2 xander staff 221K Apr 1 12:44 gandiva. cpython-37m-darwin.
>>>> so ( http://gandiva.cpython-37m-darwin.so/ )
>>>>
>>>> -rw-rw-r-- 2 xander staff 17K Jan 18 14:00 gandiva.pyx
>>>>
>>>> ```
>>>>
>>>>
>>>>
>>>> On Ubuntu, I see only the Cython file:
>>>>
>>>> ```
>>>>
>>>> $ l ~/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ | grep
>>>> gandiva
>>>>
>>>> -rw-rw-r-- 1 xander xander 17K Apr 13 12:28 gandiva.pyx
>>>>
>>>> ```
>>>>
>>>>
>>>>
>>>> Is this expected? Should I be able to import pyarrow.gandiva on Ubuntu?
>>>> Everything is run on Ubuntu so if I make use of pyarrow.gandiva I'll need
>>>> to figure out how to call it.
>>>>
>>>>
>>>>
>>>> It's mentioned here that it was removed from Python wheels but should
>>>> still be available in the conda install: https:/ / issues. apache. org/ jira/
>>>> browse/ ARROW-10154 ( https://issues.apache.org/jira/browse/ARROW-10154 ).
>>>> I'm not finding it in my Ubuntu conda install.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Xander
>>>>
>>>
>>>
>>
>>
>
>
Re: [Python] pyarrow.gandiva unavailable on Ubuntu?
Posted by Wes McKinney <we...@gmail.com>.
It looks to me like you have the wheel installed, not the conda package.
Can you reproduce this on Ubuntu from a fresh conda environment?
On Tue, Apr 13, 2021 at 3:16 PM Xander Dunn <xa...@xander.ai> wrote:
> Typo. The issue remains present. From my Ubuntu machine just now:
> ```
> $ python
> >>> import pyarrow as pa
> >>> print(pa.__file__)
> /
> home/xander/anaconda3/envs/plutus_model/lib/python3.7/site-packages/pyarrow/__init__.py
> >>> import pyarrow.plasma
> >>> import pyarrow.gandiva
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ModuleNotFoundError: No module named 'pyarrow.gandiva'
> ```
>
> The .py I'm executing on both machines is identical. Works on mac. Not
> found on Ubuntu.
>
>
> On Tue, Apr 13, 2021 at 1:01 PM, Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Hi Xander,
>> Was there autocorrect on this e-mail? the second example shows "gondiva"
>> not "gandiva"
>>
>> -Micah
>>
>> On Tue, Apr 13, 2021 at 12:59 PM Xander Dunn <xa...@xander.ai> wrote:
>>
>>> On my local macOS 11.2.3:
>>> ```
>>> $ python --version
>>> Python 3.7.10
>>> $ pip --version
>>> pip 21.0.1 from
>>> /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pip (python
>>> 3.7)
>>> $ pip list | grep pyarrow
>>> pyarrow 3.0.0
>>> $ which python
>>> /usr/local/anaconda3/envs/my_model/bin/python
>>> $ python
>>> >>> import pyarrow as pa
>>> >>> print(pa.__file__)
>>> /
>>> usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py
>>> >>> import pyarrow.plasma
>>> >>> import pyarrow.gandiva as ga
>>> >>> print(ga.__file__)
>>> /
>>> usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/gandiva.cpython-37m-darwin.so
>>> ```
>>>
>>> On my Ubuntu 14.04 instance:
>>> ```
>>> $ python --version
>>> Python 3.7.10
>>> $ pip --version
>>> pip 21.0.1 from
>>> /home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pip
>>> (python 3.7)
>>> $ pip list | grep pyarrow
>>> pyarrow 3.0.0
>>> $ which python
>>> /home/xander/anaconda3/envs/my_model/bin/python
>>> $ python
>>> >>> import pyarrow as pa
>>> >>> print(pa.__file__)
>>> /
>>> home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py
>>> >>> import pyarrow.plasma
>>> >>> import pyarrow.gondiva
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in <module>
>>> ModuleNotFoundError: No module named 'pyarrow.gondiva'
>>> ```
>>> You can see that pyarrow.gondiva is found on mac but not on Ubuntu. Same
>>> Python version. Same pyarrow version. I installed both of them with `conda
>>> install -c conda-forge pyarrow==3.0.0`.
>>>
>>> On Mac, I see the expected Cython file and library:
>>> ```
>>> $ l
>>> /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ |
>>> grep gandiva
>>> -rwxrwxr-x 2 xander staff 221K Apr 1 12:44
>>> gandiva.cpython-37m-darwin.so
>>> -rw-rw-r-- 2 xander staff 17K Jan 18 14:00 gandiva.pyx
>>> ```
>>>
>>> On Ubuntu, I see only the Cython file:
>>> ```
>>> $ l ~/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ |
>>> grep gandiva
>>> -rw-rw-r-- 1 xander xander 17K Apr 13 12:28 gandiva.pyx
>>> ```
>>>
>>> Is this expected? Should I be able to import pyarrow.gandiva on Ubuntu?
>>> Everything is run on Ubuntu so if I make use of pyarrow.gandiva I'll need
>>> to figure out how to call it.
>>>
>>> It's mentioned here that it was removed from Python wheels but should
>>> still be available in the conda install:
>>> https://issues.apache.org/jira/browse/ARROW-10154. I'm not finding it
>>> in my Ubuntu conda install.
>>>
>>> Thanks,
>>> Xander
>>>
>>
>
Re: [Python] pyarrow.gandiva unavailable on Ubuntu?
Posted by Xander Dunn <xa...@xander.ai>.
Typo. The issue remains present. From my Ubuntu machine just now:
```
$ python
>>> import pyarrow as pa
>>> print(pa.__file__)
/ home/xander/anaconda3/envs/plutus_model/lib/python3.7/site-packages/pyarrow/__init__.py ( http://home/xander/anaconda3/envs/plutus_model/lib/python3.7/site-packages/pyarrow/__init__.py )
>>> import pyarrow.plasma
>>> import pyarrow.gandiva
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pyarrow.gandiva'
```
The .py I'm executing on both machines is identical. Works on mac. Not found on Ubuntu.
On Tue, Apr 13, 2021 at 1:01 PM, Micah Kornfield < emkornfield@gmail.com > wrote:
>
> Hi Xander,
> Was there autocorrect on this e-mail? the second example shows "gondiva"
> not "gandiva"
>
>
> -Micah
>
> On Tue, Apr 13, 2021 at 12:59 PM Xander Dunn < xander@ xander. ai (
> xander@xander.ai ) > wrote:
>
>
>> On my local macOS 11.2.3:
>>
>> ```
>>
>> $ python --version
>>
>> Python 3.7.10
>>
>> $ pip --version
>>
>> pip 21.0.1 from
>> /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pip (python
>> 3.7)
>>
>> $ pip list | grep pyarrow
>>
>> pyarrow 3.0.0
>>
>> $ which python
>>
>> /usr/local/anaconda3/envs/my_model/bin/python
>>
>> $ python
>>
>> >>> import pyarrow as pa
>>
>> >>> print(pa.__file__)
>>
>> / usr/ local/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/ pyarrow/
>> __init__. py (
>> http://usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py
>> )
>>
>> >>> import pyarrow.plasma
>>
>> >>> import pyarrow.gandiva as ga
>>
>> >>> print(ga.__file__)
>>
>> / usr/ local/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/ pyarrow/
>> gandiva. cpython-37m-darwin. so (
>> http://usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/gandiva.cpython-37m-darwin.so
>> )
>>
>> ```
>>
>>
>>
>> On my Ubuntu 14.04 instance:
>>
>> ```
>>
>> $ python --version
>>
>> Python 3.7.10
>>
>> $ pip --version
>>
>> pip 21.0.1 from
>> /home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pip
>> (python 3.7)
>>
>> $ pip list | grep pyarrow
>>
>> pyarrow 3.0.0
>>
>> $ which python
>>
>> /home/xander/anaconda3/envs/my_model/bin/python
>>
>> $ python
>>
>> >>> import pyarrow as pa
>>
>> >>> print(pa.__file__)
>>
>> / home/ xander/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/
>> pyarrow/ __init__. py (
>> http://home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py
>> )
>>
>> >>> import pyarrow.plasma
>>
>> >>> import pyarrow.gondiva
>>
>> Traceback (most recent call last):
>>
>> File "<stdin>", line 1, in <module>
>>
>> ModuleNotFoundError: No module named 'pyarrow.gondiva'
>>
>> ```
>>
>> You can see that pyarrow.gondiva is found on mac but not on Ubuntu. Same
>> Python version. Same pyarrow version. I installed both of them with `conda
>> install -c conda-forge pyarrow==3.0.0`.
>>
>>
>>
>> On Mac, I see the expected Cython file and library:
>>
>> ```
>>
>> $ l
>> /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ |
>> grep gandiva
>>
>> -rwxrwxr-x 2 xander staff 221K Apr 1 12:44 gandiva. cpython-37m-darwin.
>> so ( http://gandiva.cpython-37m-darwin.so/ )
>>
>> -rw-rw-r-- 2 xander staff 17K Jan 18 14:00 gandiva.pyx
>>
>> ```
>>
>>
>>
>> On Ubuntu, I see only the Cython file:
>>
>> ```
>>
>> $ l ~/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ | grep
>> gandiva
>>
>> -rw-rw-r-- 1 xander xander 17K Apr 13 12:28 gandiva.pyx
>>
>> ```
>>
>>
>>
>> Is this expected? Should I be able to import pyarrow.gandiva on Ubuntu?
>> Everything is run on Ubuntu so if I make use of pyarrow.gandiva I'll need
>> to figure out how to call it.
>>
>>
>>
>> It's mentioned here that it was removed from Python wheels but should
>> still be available in the conda install: https:/ / issues. apache. org/ jira/
>> browse/ ARROW-10154 ( https://issues.apache.org/jira/browse/ARROW-10154 ).
>> I'm not finding it in my Ubuntu conda install.
>>
>>
>>
>> Thanks,
>>
>> Xander
>>
>>
>>
>
>
Re: [Python] pyarrow.gandiva unavailable on Ubuntu?
Posted by Micah Kornfield <em...@gmail.com>.
Hi Xander,
Was there autocorrect on this e-mail? the second example shows "gondiva"
not "gandiva"
-Micah
On Tue, Apr 13, 2021 at 12:59 PM Xander Dunn <xa...@xander.ai> wrote:
> On my local macOS 11.2.3:
> ```
> $ python --version
> Python 3.7.10
> $ pip --version
> pip 21.0.1 from
> /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pip (python
> 3.7)
> $ pip list | grep pyarrow
> pyarrow 3.0.0
> $ which python
> /usr/local/anaconda3/envs/my_model/bin/python
> $ python
> >>> import pyarrow as pa
> >>> print(pa.__file__)
> /
> usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py
> >>> import pyarrow.plasma
> >>> import pyarrow.gandiva as ga
> >>> print(ga.__file__)
> /
> usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/gandiva.cpython-37m-darwin.so
> ```
>
> On my Ubuntu 14.04 instance:
> ```
> $ python --version
> Python 3.7.10
> $ pip --version
> pip 21.0.1 from
> /home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pip
> (python 3.7)
> $ pip list | grep pyarrow
> pyarrow 3.0.0
> $ which python
> /home/xander/anaconda3/envs/my_model/bin/python
> $ python
> >>> import pyarrow as pa
> >>> print(pa.__file__)
> /
> home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py
> >>> import pyarrow.plasma
> >>> import pyarrow.gondiva
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ModuleNotFoundError: No module named 'pyarrow.gondiva'
> ```
> You can see that pyarrow.gondiva is found on mac but not on Ubuntu. Same
> Python version. Same pyarrow version. I installed both of them with `conda
> install -c conda-forge pyarrow==3.0.0`.
>
> On Mac, I see the expected Cython file and library:
> ```
> $ l
> /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ |
> grep gandiva
> -rwxrwxr-x 2 xander staff 221K Apr 1 12:44
> gandiva.cpython-37m-darwin.so
> -rw-rw-r-- 2 xander staff 17K Jan 18 14:00 gandiva.pyx
> ```
>
> On Ubuntu, I see only the Cython file:
> ```
> $ l ~/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ | grep
> gandiva
> -rw-rw-r-- 1 xander xander 17K Apr 13 12:28 gandiva.pyx
> ```
>
> Is this expected? Should I be able to import pyarrow.gandiva on Ubuntu?
> Everything is run on Ubuntu so if I make use of pyarrow.gandiva I'll need
> to figure out how to call it.
>
> It's mentioned here that it was removed from Python wheels but should
> still be available in the conda install:
> https://issues.apache.org/jira/browse/ARROW-10154. I'm not finding it in
> my Ubuntu conda install.
>
> Thanks,
> Xander
>
>