You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Alvaro-Kothe (via GitHub)" <gi...@apache.org> on 2026/04/01 22:48:36 UTC
[I] [Python] pyarrow._compute Contains AVX Instructions During Import [arrow]
Alvaro-Kothe opened a new issue, #49640:
URL: https://github.com/apache/arrow/issues/49640
### Describe the bug, including details regarding any error messages, version, and platform.
According to the Arrow [C++ environment variables documentation](https://arrow.apache.org/docs/cpp/env_vars.html#envvar-ARROW_USER_SIMD_LEVEL), the default SIMD level is SSE4.2. However, importing `pyarrow._compute` executes AVX instructions even when running on a CPU that only supports up to SSE4.2.
## Reproduction
Setup venv and install pyarrow from PyPI:
```console
$ python -m venv venv && venv/bin/pip install --no-cache pyarrow
... Successfully installed pyarrow-23.0.1
```
Then run the Python interpreter under QEMU emulating a Westmere CPU (which supports SSE4.2 but not AVX):
```console
$ qemu-x86_64-static -cpu Westmere venv/bin/python -c 'import pyarrow._compute'
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
```
The disassembly of the coredump shows that it uses AVX instructions.
```gdb
(gdb) disassemble $pc-16,$pc+16
Dump of assembler code from 0x7f2ff65ceb32 to 0x7f2ff65ceb52:
0x00007f2ff65ceb32: push r12
0x00007f2ff65ceb34: push rbp
0x00007f2ff65ceb35: mov rbp,rdi
0x00007f2ff65ceb38: push rbx
0x00007f2ff65ceb39: sub rsp,0x48
0x00007f2ff65ceb3d: call 0x7f2ff5c3c1d0
=> 0x00007f2ff65ceb42: vmovdqu xmm0,XMMWORD PTR [rax]
0x00007f2ff65ceb46: vpextrq rax,xmm0,0x1
0x00007f2ff65ceb4c: vmovdqa XMMWORD PTR [rsp+0x10],xmm0
```
Setting `ARROW_USER_SIMD_LEVEL` to `None` or `SSE4_2` didn't help.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] pyarrow._compute Contains AVX Instructions During Import [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #49640: [Python][C++] pyarrow._compute Contains AVX Instructions During Import
URL: https://github.com/apache/arrow/issues/49640
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] pyarrow._compute Contains AVX Instructions During Import [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on issue #49640:
URL: https://github.com/apache/arrow/issues/49640#issuecomment-4177833293
> It seems that we pick the flags from `/proc/cpuinfo` and that contains the host CPU flags not the target
Ho wow, I though we were using CPUID, similar to on Windows.
Fortunately there is ongoing work to fully refactor the whole CpuInfo class.
> it seems to me that is an issue under qemu.
I second that diagnosis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] pyarrow._compute Contains AVX Instructions During Import [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #49640:
URL: https://github.com/apache/arrow/issues/49640#issuecomment-4176272079
I've been able to replicate and got to this specific function registration `arrow::compute::internal::AddSumAvx2AggKernels(arrow::compute::ScalarAggregateFunction*)` from the backtrace:
```
Reading /home/raulcd/code/test-avx/lib/python3.13/site-packages/pyarrow/_compute.cpython-313-x86_64-linux-gnu.so from remote target...
[New Thread 70315.70417]
...
Thread 1 received signal SIGILL, Illegal instruction.
0x00007fda73a3cb42 in arrow::compute::internal::AddSumAvx2AggKernels(arrow::compute::ScalarAggregateFunction*) ()
from target:/home/raulcd/code/test-avx/lib/python3.13/site-packages/pyarrow/libarrow_compute.so.2300
```
The guard seems correct to me:
https://github.com/apache/arrow/blob/82847d8f6dae87e709d284b4b871da1424e93a25/cpp/src/arrow/compute/kernels/aggregate_basic.cc#L1065-L1069
It seems that we pick the flags from `/proc/cpuinfo` and that contains the host CPU flags not the target so it seems to me that is an issue under qemu.
cc @AntoinePrv @pitrou what are your thoughts?
BTW `ARROW_USER_SIMD_LEVEL` works for me:
```
$ qemu-x86_64 -cpu Westmere -L / -E ARROW_USER_SIMD_LEVEL=SSE4_2 $(which python) -c 'import pyarrow._compute; import pyarrow; print(pyarrow.__version__)'
23.0.1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] pyarrow._compute Contains AVX Instructions During Import [arrow]
Posted by "Alvaro-Kothe (via GitHub)" <gi...@apache.org>.
Alvaro-Kothe commented on issue #49640:
URL: https://github.com/apache/arrow/issues/49640#issuecomment-4700469980
Just built arrow from source on ca47cd10b651a8ff5fe44fbb23f7e01dc982a57d and I can confirm that it solved the problem. I also managed to reproduce the issue on its parent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] pyarrow._compute Contains AVX Instructions During Import [arrow]
Posted by "Alvaro-Kothe (via GitHub)" <gi...@apache.org>.
Alvaro-Kothe commented on issue #49640:
URL: https://github.com/apache/arrow/issues/49640#issuecomment-4176499931
> BTW ARROW_USER_SIMD_LEVEL works for me
Indeed, it works. I had a typo when I first tried it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] pyarrow._compute Contains AVX Instructions During Import [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #49640:
URL: https://github.com/apache/arrow/issues/49640#issuecomment-4692936106
Ok, let's close. @Alvaro-Kothe if you can still reproduce with PyArrow 25.0.0 (which should be out somewhere in July), feel free to ping here or to open a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] pyarrow._compute Contains AVX Instructions During Import [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #49640:
URL: https://github.com/apache/arrow/issues/49640#issuecomment-4682787781
@AntoinePrv Do you think this issue can be closed now that use xsimd for CPU detection?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] pyarrow._compute Contains AVX Instructions During Import [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on issue #49640:
URL: https://github.com/apache/arrow/issues/49640#issuecomment-4691493303
It should be yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org