You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Oliver Ruebenacker <ol...@broadinstitute.org> on 2023/01/06 14:59:29 UTC
[PySpark] Error using SciPy: ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
Hello,
I'm trying to install SciPy using a bootstrap script and then use it to
calculate a new field in a dataframe, running on AWS EMR.
Although the SciPy website states that only NumPy is needed, when I tried
to install SciPy using pip, pip kept failing, complaining about missing
software, until I ended up with this bootstrap script:
*sudo yum install -y python3-develsudo pip3 install -U Cythonsudo pip3
install -U pybind11sudo pip3 install -U pythransudo pip3 install -U
numpysudo pip3 install -U scipy*
At this point, the bootstrap seems to be successful, but then at this
line:
*from scipy.stats import norm*
I get the following error:
*ValueError: numpy.ndarray size changed, may indicate binary
incompatibility. Expected 88 from C header, got 80 from PyObject*
Any advice on how to proceed? Thanks!
Best, Oliver
--
Oliver Ruebenacker, Ph.D. (he)
Senior Software Engineer, Knowledge Portal Network
<http://kp4cd.org/>, Flannick
Lab <http://www.flannicklab.org/>, Broad Institute
<http://www.broadinstitute.org/>
Re: [PySpark] Error using SciPy: ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
Posted by Oliver Ruebenacker <ol...@broadinstitute.org>.
So I think now that my problem is Spark-related after all. It looks like my
bootstrap script installs SciPy just fine in a regular environment, but
somehow interaction with PySpark breaks it.
On Fri, Jan 6, 2023 at 12:39 PM Bjørn Jørgensen <bj...@gmail.com>
wrote:
> Create a Dockerfile
>
> FROM fedora
>
> RUN sudo yum install -y python3-devel
> RUN sudo pip3 install -U Cython && \
> sudo pip3 install -U pybind11 && \
> sudo pip3 install -U pythran && \
> sudo pip3 install -U numpy && \
> sudo pip3 install -U scipy
>
>
>
>
>
>
> docker build --pull --rm -f "Dockerfile" -t fedoratest:latest "."
>
> Sending build context to Docker daemon 2.048kB
> Step 1/3 : FROM fedora
> latest: Pulling from library/fedora
> Digest:
> sha256:3487c98481d1bba7e769cf7bcecd6343c2d383fdd6bed34ec541b6b23ef07664
> Status: Image is up to date for fedora:latest
> ---> 95b7a2603d3a
> Step 2/3 : RUN sudo yum install -y python3-devel
> ---> Running in a7c648ae7014
> Fedora 37 - x86_64 7.5 MB/s | 64 MB
> 00:08
> Fedora 37 openh264 (From Cisco) - x86_64 418 B/s | 2.5 kB
> 00:06
> Fedora Modular 37 - x86_64 471 kB/s | 3.0 MB
> 00:06
> Fedora 37 - x86_64 - Updates 3.0 MB/s | 20 MB
> 00:06
> Fedora Modular 37 - x86_64 - Updates 179 kB/s | 1.1 MB
> 00:06
> Last metadata expiration check: 0:00:01 ago on Fri Jan 6 17:37:59 2023.
> Dependencies resolved.
>
> ================================================================================
> Package Architecture Version Repository
> Size
>
> ================================================================================
> Installing:
> python3-devel x86_64 3.11.1-1.fc37 updates
> 269 k
> Upgrading:
> python3 x86_64 3.11.1-1.fc37 updates
> 27 k
> python3-libs x86_64 3.11.1-1.fc37 updates
> 9.6 M
> Installing dependencies:
> libpkgconf x86_64 1.8.0-3.fc37 fedora
> 36 k
> pkgconf x86_64 1.8.0-3.fc37 fedora
> 41 k
> pkgconf-m4 noarch 1.8.0-3.fc37 fedora
> 14 k
> pkgconf-pkg-config x86_64 1.8.0-3.fc37 fedora
> 10 k
> Installing weak dependencies:
> python3-pip noarch 22.2.2-3.fc37 updates
> 3.1 M
> python3-setuptools noarch 62.6.0-2.fc37 fedora
> 1.6 M
>
> Transaction Summary
>
> ================================================================================
> Install 7 Packages
> Upgrade 2 Packages
>
> Total download size: 15 M
> Downloading Packages:
> (1/9): pkgconf-m4-1.8.0-3.fc37.noarch.rpm 2.9 kB/s | 14 kB
> 00:05
> (2/9): libpkgconf-1.8.0-3.fc37.x86_64.rpm 7.1 kB/s | 36 kB
> 00:05
> (3/9): pkgconf-1.8.0-3.fc37.x86_64.rpm 8.2 kB/s | 41 kB
> 00:05
> (4/9): pkgconf-pkg-config-1.8.0-3.fc37.x86_64.r 143 kB/s | 10 kB
> 00:00
> (5/9): python3-devel-3.11.1-1.fc37.x86_64.rpm 458 kB/s | 269 kB
> 00:00
> (6/9): python3-3.11.1-1.fc37.x86_64.rpm 442 kB/s | 27 kB
> 00:00
> (7/9): python3-setuptools-62.6.0-2.fc37.noarch. 2.1 MB/s | 1.6 MB
> 00:00
> (8/9): python3-pip-22.2.2-3.fc37.noarch.rpm 4.0 MB/s | 3.1 MB
> 00:00
> (9/9): python3-libs-3.11.1-1.fc37.x86_64.rpm 7.2 MB/s | 9.6 MB
> 00:01
>
> --------------------------------------------------------------------------------
> Total 1.8 MB/s | 15 MB
> 00:08
> Running transaction check
> Transaction check succeeded.
> Running transaction test
> Transaction test succeeded.
> Running transaction
> Preparing :
> 1/1
> Upgrading : python3-libs-3.11.1-1.fc37.x86_64
> 1/11
> Upgrading : python3-3.11.1-1.fc37.x86_64
> 2/11
> Installing : python3-setuptools-62.6.0-2.fc37.noarch
> 3/11
> Installing : python3-pip-22.2.2-3.fc37.noarch
> 4/11
> Installing : pkgconf-m4-1.8.0-3.fc37.noarch
> 5/11
> Installing : libpkgconf-1.8.0-3.fc37.x86_64
> 6/11
> Installing : pkgconf-1.8.0-3.fc37.x86_64
> 7/11
> Installing : pkgconf-pkg-config-1.8.0-3.fc37.x86_64
> 8/11
> Installing : python3-devel-3.11.1-1.fc37.x86_64
> 9/11
> Cleanup : python3-3.11.0-1.fc37.x86_64
> 10/11
> Cleanup : python3-libs-3.11.0-1.fc37.x86_64
> 11/11
> Running scriptlet: python3-libs-3.11.0-1.fc37.x86_64
> 11/11
> Verifying : libpkgconf-1.8.0-3.fc37.x86_64
> 1/11
> Verifying : pkgconf-1.8.0-3.fc37.x86_64
> 2/11
> Verifying : pkgconf-m4-1.8.0-3.fc37.noarch
> 3/11
> Verifying : pkgconf-pkg-config-1.8.0-3.fc37.x86_64
> 4/11
> Verifying : python3-setuptools-62.6.0-2.fc37.noarch
> 5/11
> Verifying : python3-devel-3.11.1-1.fc37.x86_64
> 6/11
> Verifying : python3-pip-22.2.2-3.fc37.noarch
> 7/11
> Verifying : python3-3.11.1-1.fc37.x86_64
> 8/11
> Verifying : python3-3.11.0-1.fc37.x86_64
> 9/11
> Verifying : python3-libs-3.11.1-1.fc37.x86_64
> 10/11
> Verifying : python3-libs-3.11.0-1.fc37.x86_64
> 11/11
>
> Upgraded:
> python3-3.11.1-1.fc37.x86_64 python3-libs-3.11.1-1.fc37.x86_64
>
> Installed:
> libpkgconf-1.8.0-3.fc37.x86_64
>
> pkgconf-1.8.0-3.fc37.x86_64
>
> pkgconf-m4-1.8.0-3.fc37.noarch
>
> pkgconf-pkg-config-1.8.0-3.fc37.x86_64
>
> python3-devel-3.11.1-1.fc37.x86_64
>
> python3-pip-22.2.2-3.fc37.noarch
>
> python3-setuptools-62.6.0-2.fc37.noarch
>
>
> Complete!
> Removing intermediate container a7c648ae7014
> ---> 593ffbdb0c08
> Step 3/3 : RUN sudo pip3 install -U Cython && sudo pip3 install -U
> pybind11 && sudo pip3 install -U pythran && sudo pip3 install -U numpy &&
> sudo pip3 install -U scipy
> ---> Running in 97aa856851b4
> Collecting Cython
> Downloading
> Cython-0.29.33-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
> (1.9 MB)
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 21.0 MB/s eta
> 0:00:00
> Installing collected packages: Cython
> Successfully installed Cython-0.29.33
> WARNING: Running pip as the 'root' user can result in broken permissions
> and conflicting behaviour with the system package manager. It is
> recommended to use a virtual environment instead:
> https://pip.pypa.io/warnings/venv
> Collecting pybind11
> Downloading pybind11-2.10.3-py3-none-any.whl (222 kB)
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 222.4/222.4 kB 4.4 MB/s eta
> 0:00:00
> Installing collected packages: pybind11
> Successfully installed pybind11-2.10.3
> WARNING: Running pip as the 'root' user can result in broken permissions
> and conflicting behaviour with the system package manager. It is
> recommended to use a virtual environment instead:
> https://pip.pypa.io/warnings/venv
> Collecting pythran
> Downloading pythran-0.12.0-py3-none-any.whl (4.2 MB)
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.2/4.2 MB 26.8 MB/s eta
> 0:00:00
> Collecting ply>=3.4
> Downloading ply-3.11-py2.py3-none-any.whl (49 kB)
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.6/49.6 kB 7.1 MB/s eta
> 0:00:00
> Collecting gast~=0.5.0
> Downloading gast-0.5.3-py3-none-any.whl (19 kB)
> Collecting numpy
> Downloading
> numpy-1.24.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
> (17.3 MB)
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 34.8 MB/s eta
> 0:00:00
> Collecting beniget~=0.4.0
> Downloading beniget-0.4.1-py3-none-any.whl (9.4 kB)
> Installing collected packages: ply, numpy, gast, beniget, pythran
> Successfully installed beniget-0.4.1 gast-0.5.3 numpy-1.24.1 ply-3.11
> pythran-0.12.0
> WARNING: Running pip as the 'root' user can result in broken permissions
> and conflicting behaviour with the system package manager. It is
> recommended to use a virtual environment instead:
> https://pip.pypa.io/warnings/venv
> Requirement already satisfied: numpy in
> /usr/local/lib64/python3.11/site-packages (1.24.1)
> WARNING: Running pip as the 'root' user can result in broken permissions
> and conflicting behaviour with the system package manager. It is
> recommended to use a virtual environment instead:
> https://pip.pypa.io/warnings/venv
> Collecting scipy
> Downloading
> scipy-1.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
> (34.1 MB)
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.1/34.1 MB 30.8 MB/s eta
> 0:00:00
> Requirement already satisfied: numpy<1.27.0,>=1.19.5 in
> /usr/local/lib64/python3.11/site-packages (from scipy) (1.24.1)
> Installing collected packages: scipy
> Successfully installed scipy-1.10.0
> WARNING: Running pip as the 'root' user can result in broken permissions
> and conflicting behaviour with the system package manager. It is
> recommended to use a virtual environment instead:
> https://pip.pypa.io/warnings/venv
> Removing intermediate container 97aa856851b4
> ---> 9b2a51c94efe
> Successfully built 9b2a51c94efe
> Successfully tagged fedoratest:latest
>
>
>
>
> [root@afebdea35f7c /]# python
> bash: python: command not found
> [root@afebdea35f7c /]# python3
> Python 3.11.1 (main, Dec 7 2022, 00:00:00) [GCC 12.2.1 20221121 (Red Hat
> 12.2.1-4)] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from scipy.stats import norm
> >>>
>
>
>
> fre. 6. jan. 2023 kl. 18:12 skrev Oliver Ruebenacker <
> oliverr@broadinstitute.org>:
>
>> Thank you for the link. I already tried most of what was suggested there,
>> but without success.
>>
>> On Fri, Jan 6, 2023 at 11:35 AM Bjørn Jørgensen <bj...@gmail.com>
>> wrote:
>>
>>>
>>>
>>>
>>> https://stackoverflow.com/questions/66060487/valueerror-numpy-ndarray-size-changed-may-indicate-binary-incompatibility-exp
>>>
>>>
>>>
>>>
>>> fre. 6. jan. 2023, 16:01 skrev Oliver Ruebenacker <
>>> oliverr@broadinstitute.org>:
>>>
>>>>
>>>> Hello,
>>>>
>>>> I'm trying to install SciPy using a bootstrap script and then use it
>>>> to calculate a new field in a dataframe, running on AWS EMR.
>>>>
>>>> Although the SciPy website states that only NumPy is needed, when I
>>>> tried to install SciPy using pip, pip kept failing, complaining about
>>>> missing software, until I ended up with this bootstrap script:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *sudo yum install -y python3-develsudo pip3 install -U Cythonsudo pip3
>>>> install -U pybind11sudo pip3 install -U pythransudo pip3 install -U
>>>> numpysudo pip3 install -U scipy*
>>>>
>>>> At this point, the bootstrap seems to be successful, but then at this
>>>> line:
>>>>
>>>> *from scipy.stats import norm*
>>>>
>>>> I get the following error:
>>>>
>>>> *ValueError: numpy.ndarray size changed, may indicate binary
>>>> incompatibility. Expected 88 from C header, got 80 from PyObject*
>>>>
>>>> Any advice on how to proceed? Thanks!
>>>>
>>>> Best, Oliver
>>>>
>>>> --
>>>> Oliver Ruebenacker, Ph.D. (he)
>>>> Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>,
>>>> Flannick Lab <http://www.flannicklab.org/>, Broad Institute
>>>> <http://www.broadinstitute.org/>
>>>>
>>>
>>
>> --
>> Oliver Ruebenacker, Ph.D. (he)
>> Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>, Flannick
>> Lab <http://www.flannicklab.org/>, Broad Institute
>> <http://www.broadinstitute.org/>
>>
>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>
--
Oliver Ruebenacker, Ph.D. (he)
Senior Software Engineer, Knowledge Portal Network
<http://kp4cd.org/>, Flannick
Lab <http://www.flannicklab.org/>, Broad Institute
<http://www.broadinstitute.org/>
Re: [PySpark] Error using SciPy: ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
Posted by Bjørn Jørgensen <bj...@gmail.com>.
Create a Dockerfile
FROM fedora
RUN sudo yum install -y python3-devel
RUN sudo pip3 install -U Cython && \
sudo pip3 install -U pybind11 && \
sudo pip3 install -U pythran && \
sudo pip3 install -U numpy && \
sudo pip3 install -U scipy
docker build --pull --rm -f "Dockerfile" -t fedoratest:latest "."
Sending build context to Docker daemon 2.048kB
Step 1/3 : FROM fedora
latest: Pulling from library/fedora
Digest:
sha256:3487c98481d1bba7e769cf7bcecd6343c2d383fdd6bed34ec541b6b23ef07664
Status: Image is up to date for fedora:latest
---> 95b7a2603d3a
Step 2/3 : RUN sudo yum install -y python3-devel
---> Running in a7c648ae7014
Fedora 37 - x86_64 7.5 MB/s | 64 MB 00:08
Fedora 37 openh264 (From Cisco) - x86_64 418 B/s | 2.5 kB 00:06
Fedora Modular 37 - x86_64 471 kB/s | 3.0 MB 00:06
Fedora 37 - x86_64 - Updates 3.0 MB/s | 20 MB 00:06
Fedora Modular 37 - x86_64 - Updates 179 kB/s | 1.1 MB 00:06
Last metadata expiration check: 0:00:01 ago on Fri Jan 6 17:37:59 2023.
Dependencies resolved.
================================================================================
Package Architecture Version Repository
Size
================================================================================
Installing:
python3-devel x86_64 3.11.1-1.fc37 updates
269 k
Upgrading:
python3 x86_64 3.11.1-1.fc37 updates
27 k
python3-libs x86_64 3.11.1-1.fc37 updates
9.6 M
Installing dependencies:
libpkgconf x86_64 1.8.0-3.fc37 fedora
36 k
pkgconf x86_64 1.8.0-3.fc37 fedora
41 k
pkgconf-m4 noarch 1.8.0-3.fc37 fedora
14 k
pkgconf-pkg-config x86_64 1.8.0-3.fc37 fedora
10 k
Installing weak dependencies:
python3-pip noarch 22.2.2-3.fc37 updates
3.1 M
python3-setuptools noarch 62.6.0-2.fc37 fedora
1.6 M
Transaction Summary
================================================================================
Install 7 Packages
Upgrade 2 Packages
Total download size: 15 M
Downloading Packages:
(1/9): pkgconf-m4-1.8.0-3.fc37.noarch.rpm 2.9 kB/s | 14 kB 00:05
(2/9): libpkgconf-1.8.0-3.fc37.x86_64.rpm 7.1 kB/s | 36 kB 00:05
(3/9): pkgconf-1.8.0-3.fc37.x86_64.rpm 8.2 kB/s | 41 kB 00:05
(4/9): pkgconf-pkg-config-1.8.0-3.fc37.x86_64.r 143 kB/s | 10 kB 00:00
(5/9): python3-devel-3.11.1-1.fc37.x86_64.rpm 458 kB/s | 269 kB 00:00
(6/9): python3-3.11.1-1.fc37.x86_64.rpm 442 kB/s | 27 kB 00:00
(7/9): python3-setuptools-62.6.0-2.fc37.noarch. 2.1 MB/s | 1.6 MB 00:00
(8/9): python3-pip-22.2.2-3.fc37.noarch.rpm 4.0 MB/s | 3.1 MB 00:00
(9/9): python3-libs-3.11.1-1.fc37.x86_64.rpm 7.2 MB/s | 9.6 MB 00:01
--------------------------------------------------------------------------------
Total 1.8 MB/s | 15 MB 00:08
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing :
1/1
Upgrading : python3-libs-3.11.1-1.fc37.x86_64
1/11
Upgrading : python3-3.11.1-1.fc37.x86_64
2/11
Installing : python3-setuptools-62.6.0-2.fc37.noarch
3/11
Installing : python3-pip-22.2.2-3.fc37.noarch
4/11
Installing : pkgconf-m4-1.8.0-3.fc37.noarch
5/11
Installing : libpkgconf-1.8.0-3.fc37.x86_64
6/11
Installing : pkgconf-1.8.0-3.fc37.x86_64
7/11
Installing : pkgconf-pkg-config-1.8.0-3.fc37.x86_64
8/11
Installing : python3-devel-3.11.1-1.fc37.x86_64
9/11
Cleanup : python3-3.11.0-1.fc37.x86_64
10/11
Cleanup : python3-libs-3.11.0-1.fc37.x86_64
11/11
Running scriptlet: python3-libs-3.11.0-1.fc37.x86_64
11/11
Verifying : libpkgconf-1.8.0-3.fc37.x86_64
1/11
Verifying : pkgconf-1.8.0-3.fc37.x86_64
2/11
Verifying : pkgconf-m4-1.8.0-3.fc37.noarch
3/11
Verifying : pkgconf-pkg-config-1.8.0-3.fc37.x86_64
4/11
Verifying : python3-setuptools-62.6.0-2.fc37.noarch
5/11
Verifying : python3-devel-3.11.1-1.fc37.x86_64
6/11
Verifying : python3-pip-22.2.2-3.fc37.noarch
7/11
Verifying : python3-3.11.1-1.fc37.x86_64
8/11
Verifying : python3-3.11.0-1.fc37.x86_64
9/11
Verifying : python3-libs-3.11.1-1.fc37.x86_64
10/11
Verifying : python3-libs-3.11.0-1.fc37.x86_64
11/11
Upgraded:
python3-3.11.1-1.fc37.x86_64 python3-libs-3.11.1-1.fc37.x86_64
Installed:
libpkgconf-1.8.0-3.fc37.x86_64
pkgconf-1.8.0-3.fc37.x86_64
pkgconf-m4-1.8.0-3.fc37.noarch
pkgconf-pkg-config-1.8.0-3.fc37.x86_64
python3-devel-3.11.1-1.fc37.x86_64
python3-pip-22.2.2-3.fc37.noarch
python3-setuptools-62.6.0-2.fc37.noarch
Complete!
Removing intermediate container a7c648ae7014
---> 593ffbdb0c08
Step 3/3 : RUN sudo pip3 install -U Cython && sudo pip3 install -U pybind11
&& sudo pip3 install -U pythran && sudo pip3 install -U numpy && sudo pip3
install -U scipy
---> Running in 97aa856851b4
Collecting Cython
Downloading
Cython-0.29.33-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
(1.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 21.0 MB/s eta
0:00:00
Installing collected packages: Cython
Successfully installed Cython-0.29.33
WARNING: Running pip as the 'root' user can result in broken permissions
and conflicting behaviour with the system package manager. It is
recommended to use a virtual environment instead:
https://pip.pypa.io/warnings/venv
Collecting pybind11
Downloading pybind11-2.10.3-py3-none-any.whl (222 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 222.4/222.4 kB 4.4 MB/s eta
0:00:00
Installing collected packages: pybind11
Successfully installed pybind11-2.10.3
WARNING: Running pip as the 'root' user can result in broken permissions
and conflicting behaviour with the system package manager. It is
recommended to use a virtual environment instead:
https://pip.pypa.io/warnings/venv
Collecting pythran
Downloading pythran-0.12.0-py3-none-any.whl (4.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.2/4.2 MB 26.8 MB/s eta
0:00:00
Collecting ply>=3.4
Downloading ply-3.11-py2.py3-none-any.whl (49 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.6/49.6 kB 7.1 MB/s eta
0:00:00
Collecting gast~=0.5.0
Downloading gast-0.5.3-py3-none-any.whl (19 kB)
Collecting numpy
Downloading
numpy-1.24.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
(17.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 34.8 MB/s eta
0:00:00
Collecting beniget~=0.4.0
Downloading beniget-0.4.1-py3-none-any.whl (9.4 kB)
Installing collected packages: ply, numpy, gast, beniget, pythran
Successfully installed beniget-0.4.1 gast-0.5.3 numpy-1.24.1 ply-3.11
pythran-0.12.0
WARNING: Running pip as the 'root' user can result in broken permissions
and conflicting behaviour with the system package manager. It is
recommended to use a virtual environment instead:
https://pip.pypa.io/warnings/venv
Requirement already satisfied: numpy in
/usr/local/lib64/python3.11/site-packages (1.24.1)
WARNING: Running pip as the 'root' user can result in broken permissions
and conflicting behaviour with the system package manager. It is
recommended to use a virtual environment instead:
https://pip.pypa.io/warnings/venv
Collecting scipy
Downloading
scipy-1.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
(34.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.1/34.1 MB 30.8 MB/s eta
0:00:00
Requirement already satisfied: numpy<1.27.0,>=1.19.5 in
/usr/local/lib64/python3.11/site-packages (from scipy) (1.24.1)
Installing collected packages: scipy
Successfully installed scipy-1.10.0
WARNING: Running pip as the 'root' user can result in broken permissions
and conflicting behaviour with the system package manager. It is
recommended to use a virtual environment instead:
https://pip.pypa.io/warnings/venv
Removing intermediate container 97aa856851b4
---> 9b2a51c94efe
Successfully built 9b2a51c94efe
Successfully tagged fedoratest:latest
[root@afebdea35f7c /]# python
bash: python: command not found
[root@afebdea35f7c /]# python3
Python 3.11.1 (main, Dec 7 2022, 00:00:00) [GCC 12.2.1 20221121 (Red Hat
12.2.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from scipy.stats import norm
>>>
fre. 6. jan. 2023 kl. 18:12 skrev Oliver Ruebenacker <
oliverr@broadinstitute.org>:
> Thank you for the link. I already tried most of what was suggested there,
> but without success.
>
> On Fri, Jan 6, 2023 at 11:35 AM Bjørn Jørgensen <bj...@gmail.com>
> wrote:
>
>>
>>
>>
>> https://stackoverflow.com/questions/66060487/valueerror-numpy-ndarray-size-changed-may-indicate-binary-incompatibility-exp
>>
>>
>>
>>
>> fre. 6. jan. 2023, 16:01 skrev Oliver Ruebenacker <
>> oliverr@broadinstitute.org>:
>>
>>>
>>> Hello,
>>>
>>> I'm trying to install SciPy using a bootstrap script and then use it
>>> to calculate a new field in a dataframe, running on AWS EMR.
>>>
>>> Although the SciPy website states that only NumPy is needed, when I
>>> tried to install SciPy using pip, pip kept failing, complaining about
>>> missing software, until I ended up with this bootstrap script:
>>>
>>>
>>>
>>>
>>>
>>>
>>> *sudo yum install -y python3-develsudo pip3 install -U Cythonsudo pip3
>>> install -U pybind11sudo pip3 install -U pythransudo pip3 install -U
>>> numpysudo pip3 install -U scipy*
>>>
>>> At this point, the bootstrap seems to be successful, but then at this
>>> line:
>>>
>>> *from scipy.stats import norm*
>>>
>>> I get the following error:
>>>
>>> *ValueError: numpy.ndarray size changed, may indicate binary
>>> incompatibility. Expected 88 from C header, got 80 from PyObject*
>>>
>>> Any advice on how to proceed? Thanks!
>>>
>>> Best, Oliver
>>>
>>> --
>>> Oliver Ruebenacker, Ph.D. (he)
>>> Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>, Flannick
>>> Lab <http://www.flannicklab.org/>, Broad Institute
>>> <http://www.broadinstitute.org/>
>>>
>>
>
> --
> Oliver Ruebenacker, Ph.D. (he)
> Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>, Flannick
> Lab <http://www.flannicklab.org/>, Broad Institute
> <http://www.broadinstitute.org/>
>
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: [PySpark] Error using SciPy: ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
Posted by Mich Talebzadeh <mi...@gmail.com>.
https://stackoverflow.com/questions/66060487/valueerror-numpy-ndarray-size-changed-may-indicate-binary-incompatibility-exp
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Fri, 6 Jan 2023 at 17:13, Oliver Ruebenacker <ol...@broadinstitute.org>
wrote:
> Thank you for the link. I already tried most of what was suggested there,
> but without success.
>
> On Fri, Jan 6, 2023 at 11:35 AM Bjørn Jørgensen <bj...@gmail.com>
> wrote:
>
>>
>>
>>
>> https://stackoverflow.com/questions/66060487/valueerror-numpy-ndarray-size-changed-may-indicate-binary-incompatibility-exp
>>
>>
>>
>>
>> fre. 6. jan. 2023, 16:01 skrev Oliver Ruebenacker <
>> oliverr@broadinstitute.org>:
>>
>>>
>>> Hello,
>>>
>>> I'm trying to install SciPy using a bootstrap script and then use it
>>> to calculate a new field in a dataframe, running on AWS EMR.
>>>
>>> Although the SciPy website states that only NumPy is needed, when I
>>> tried to install SciPy using pip, pip kept failing, complaining about
>>> missing software, until I ended up with this bootstrap script:
>>>
>>>
>>>
>>>
>>>
>>>
>>> *sudo yum install -y python3-develsudo pip3 install -U Cythonsudo pip3
>>> install -U pybind11sudo pip3 install -U pythransudo pip3 install -U
>>> numpysudo pip3 install -U scipy*
>>>
>>> At this point, the bootstrap seems to be successful, but then at this
>>> line:
>>>
>>> *from scipy.stats import norm*
>>>
>>> I get the following error:
>>>
>>> *ValueError: numpy.ndarray size changed, may indicate binary
>>> incompatibility. Expected 88 from C header, got 80 from PyObject*
>>>
>>> Any advice on how to proceed? Thanks!
>>>
>>> Best, Oliver
>>>
>>> --
>>> Oliver Ruebenacker, Ph.D. (he)
>>> Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>, Flannick
>>> Lab <http://www.flannicklab.org/>, Broad Institute
>>> <http://www.broadinstitute.org/>
>>>
>>
>
> --
> Oliver Ruebenacker, Ph.D. (he)
> Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>, Flannick
> Lab <http://www.flannicklab.org/>, Broad Institute
> <http://www.broadinstitute.org/>
>
Re: [PySpark] Error using SciPy: ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
Posted by Oliver Ruebenacker <ol...@broadinstitute.org>.
Thank you for the link. I already tried most of what was suggested there,
but without success.
On Fri, Jan 6, 2023 at 11:35 AM Bjørn Jørgensen <bj...@gmail.com>
wrote:
>
>
>
> https://stackoverflow.com/questions/66060487/valueerror-numpy-ndarray-size-changed-may-indicate-binary-incompatibility-exp
>
>
>
>
> fre. 6. jan. 2023, 16:01 skrev Oliver Ruebenacker <
> oliverr@broadinstitute.org>:
>
>>
>> Hello,
>>
>> I'm trying to install SciPy using a bootstrap script and then use it to
>> calculate a new field in a dataframe, running on AWS EMR.
>>
>> Although the SciPy website states that only NumPy is needed, when I
>> tried to install SciPy using pip, pip kept failing, complaining about
>> missing software, until I ended up with this bootstrap script:
>>
>>
>>
>>
>>
>>
>> *sudo yum install -y python3-develsudo pip3 install -U Cythonsudo pip3
>> install -U pybind11sudo pip3 install -U pythransudo pip3 install -U
>> numpysudo pip3 install -U scipy*
>>
>> At this point, the bootstrap seems to be successful, but then at this
>> line:
>>
>> *from scipy.stats import norm*
>>
>> I get the following error:
>>
>> *ValueError: numpy.ndarray size changed, may indicate binary
>> incompatibility. Expected 88 from C header, got 80 from PyObject*
>>
>> Any advice on how to proceed? Thanks!
>>
>> Best, Oliver
>>
>> --
>> Oliver Ruebenacker, Ph.D. (he)
>> Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>, Flannick
>> Lab <http://www.flannicklab.org/>, Broad Institute
>> <http://www.broadinstitute.org/>
>>
>
--
Oliver Ruebenacker, Ph.D. (he)
Senior Software Engineer, Knowledge Portal Network
<http://kp4cd.org/>, Flannick
Lab <http://www.flannicklab.org/>, Broad Institute
<http://www.broadinstitute.org/>
Re: [PySpark] Error using SciPy: ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
Posted by Bjørn Jørgensen <bj...@gmail.com>.
https://stackoverflow.com/questions/66060487/valueerror-numpy-ndarray-size-changed-may-indicate-binary-incompatibility-exp
fre. 6. jan. 2023, 16:01 skrev Oliver Ruebenacker <
oliverr@broadinstitute.org>:
>
> Hello,
>
> I'm trying to install SciPy using a bootstrap script and then use it to
> calculate a new field in a dataframe, running on AWS EMR.
>
> Although the SciPy website states that only NumPy is needed, when I
> tried to install SciPy using pip, pip kept failing, complaining about
> missing software, until I ended up with this bootstrap script:
>
>
>
>
>
>
> *sudo yum install -y python3-develsudo pip3 install -U Cythonsudo pip3
> install -U pybind11sudo pip3 install -U pythransudo pip3 install -U
> numpysudo pip3 install -U scipy*
>
> At this point, the bootstrap seems to be successful, but then at this
> line:
>
> *from scipy.stats import norm*
>
> I get the following error:
>
> *ValueError: numpy.ndarray size changed, may indicate binary
> incompatibility. Expected 88 from C header, got 80 from PyObject*
>
> Any advice on how to proceed? Thanks!
>
> Best, Oliver
>
> --
> Oliver Ruebenacker, Ph.D. (he)
> Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>, Flannick
> Lab <http://www.flannicklab.org/>, Broad Institute
> <http://www.broadinstitute.org/>
>