You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Nemecsek (via GitHub)" <gi...@apache.org> on 2023/03/10 08:51:42 UTC
[GitHub] [arrow] Nemecsek opened a new issue, #34527: pyarrow crashes Python on import on AMD CPUs
Nemecsek opened a new issue, #34527:
URL: https://github.com/apache/arrow/issues/34527
### Describe the bug, including details regarding any error messages, version, and platform.
`pyarrow 11.0.0` (and some previous versions) doesn't work on some AMD x86_64 CPUs such as AMD G-T40 and AMD GX-412TC.
`import pyarrow` crashes python with an error `Invalid machine instruction` (my free translation from `Ungültiger Maschinenbefehl` in German).
It is due to a **SSE feature** that is not available on the CPUs and is requested by `pyarrow` on import.
According to the [documentation](https://arrow.apache.org/docs/cpp/env_vars.html#cpp-env-vars) a possible solution is to set environment var `ARROW_USER_SIMD_LEVEL=NONE`. It looks like the initial check returns SSE is available for these AMD CPUs even if it is not:
> **ARROW_USER_SIMD_LEVEL**
The SIMD optimization level to select. By default, Arrow C++ detects the capabilities of the current CPU at runtime and chooses the best execution paths based on that information. One can override the detection by setting this environment variable to a well-defined value. Supported values are:
> * **NONE** disables any runtime-selected SIMD
> * **SSE4_2** enables any SSE2-based optimizations until SSE4.2 (
> * **AVX** enables any AVX-based optimizations and earlier;
> * **AVX2** enables any AVX2-based optimizations and earlier;
> * **AVX512** enables any AVX512-based optimizations and earlier.
> This environment variable only has an effect on x86 platforms.
[Here](https://arrow.apache.org/docs/python/env_vars.html) it is stated that
> Since PyArrow is based on Arrow C++, its behavior is also affected by the [Arrow C++ environment variables](https://arrow.apache.org/docs/cpp/env_vars.html#cpp-env-vars).
After a number of attempts, it doesn't work: `ARROW_USER_SIMD_LEVEL` is neglected, and the only solution looks like to be a lenghty recompilation of `pyarrow`.
Am I missing something?
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Nemecsek commented on issue #34527: pyarrow crashes Python on import on AMD CPUs
Posted by "Nemecsek (via GitHub)" <gi...@apache.org>.
Nemecsek commented on issue #34527:
URL: https://github.com/apache/arrow/issues/34527#issuecomment-1463513582
@cyb70289, it is what I did as it was the only practical solution after hours of testing.
But why is the environment var `ARROW_USER_SIMD_LEVEL` not read?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on issue #34527: pyarrow crashes Python on import on AMD CPUs
Posted by "cyb70289 (via GitHub)" <gi...@apache.org>.
cyb70289 commented on issue #34527:
URL: https://github.com/apache/arrow/issues/34527#issuecomment-1463508581
You have to rebuild pyarrow with sse disabled. See https://github.com/apache/arrow/issues/34268 and https://github.com/apache/arrow/issues/34277.
@abcbarryn shared the build steps https://github.com/apache/arrow/issues/34277#issuecomment-1449455571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on issue #34527: pyarrow crashes Python on import on AMD CPUs
Posted by "cyb70289 (via GitHub)" <gi...@apache.org>.
cyb70289 commented on issue #34527:
URL: https://github.com/apache/arrow/issues/34527#issuecomment-1463541818
> It looks like I am not the only one who has fallen into this trap.
>
Indeed, we do need to improve the document about this SIGILL issue. Will open a bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Nemecsek commented on issue #34527: pyarrow crashes Python on import on AMD CPUs
Posted by "Nemecsek (via GitHub)" <gi...@apache.org>.
Nemecsek commented on issue #34527:
URL: https://github.com/apache/arrow/issues/34527#issuecomment-1463530159
@cyb70289 , thank you for your explanation.
It looks like I am not the only one who has fallen into this trap.
It is confusing for a non-expert like me (I want to install `streamlit`, that will then import `pyarrow` together with another ton of libraries) to find documentation that doesn't match the experience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on issue #34527: pyarrow crashes Python on import on AMD CPUs
Posted by "cyb70289 (via GitHub)" <gi...@apache.org>.
cyb70289 commented on issue #34527:
URL: https://github.com/apache/arrow/issues/34527#issuecomment-1463520789
I explained it at https://github.com/apache/arrow/issues/34268#issuecomment-1439359645
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Nemecsek closed issue #34527: pyarrow crashes Python on import on AMD CPUs
Posted by "Nemecsek (via GitHub)" <gi...@apache.org>.
Nemecsek closed issue #34527: pyarrow crashes Python on import on AMD CPUs
URL: https://github.com/apache/arrow/issues/34527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org