You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Alvaro-Kothe (via GitHub)" <gi...@apache.org> on 2026/04/01 22:48:36 UTC

[I] [Python] pyarrow._compute Contains AVX Instructions During Import [arrow]

Alvaro-Kothe opened a new issue, #49640:
URL: https://github.com/apache/arrow/issues/49640

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   According to the Arrow [C++ environment variables documentation](https://arrow.apache.org/docs/cpp/env_vars.html#envvar-ARROW_USER_SIMD_LEVEL), the default SIMD level is SSE4.2. However, importing `pyarrow._compute` executes AVX instructions even when running on a CPU that only supports up to SSE4.2.
   
   ## Reproduction
   
   Setup venv and install pyarrow from PyPI:
   ```console
   $ python -m venv venv && venv/bin/pip install --no-cache pyarrow
   ... Successfully installed pyarrow-23.0.1
   ```
   
   Then run the Python interpreter under QEMU emulating a Westmere CPU (which supports SSE4.2 but not AVX):
   ```console
   $ qemu-x86_64-static -cpu Westmere venv/bin/python -c 'import pyarrow._compute'
   qemu: uncaught target signal 4 (Illegal instruction) - core dumped
   ```
   
   The disassembly of the coredump shows that it uses AVX instructions.
   
   ```gdb
   (gdb) disassemble $pc-16,$pc+16
   Dump of assembler code from 0x7f2ff65ceb32 to 0x7f2ff65ceb52:
      0x00007f2ff65ceb32:  push   r12
      0x00007f2ff65ceb34:  push   rbp
      0x00007f2ff65ceb35:  mov    rbp,rdi
      0x00007f2ff65ceb38:  push   rbx
      0x00007f2ff65ceb39:  sub    rsp,0x48
      0x00007f2ff65ceb3d:  call   0x7f2ff5c3c1d0
   => 0x00007f2ff65ceb42:  vmovdqu xmm0,XMMWORD PTR [rax]
      0x00007f2ff65ceb46:  vpextrq rax,xmm0,0x1
      0x00007f2ff65ceb4c:  vmovdqa XMMWORD PTR [rsp+0x10],xmm0
   ```
   
   Setting `ARROW_USER_SIMD_LEVEL` to `None` or `SSE4_2` didn't help.
   
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python][C++] pyarrow._compute Contains AVX Instructions During Import [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #49640: [Python][C++] pyarrow._compute Contains AVX Instructions During Import
URL: https://github.com/apache/arrow/issues/49640


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python][C++] pyarrow._compute Contains AVX Instructions During Import [arrow]

Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on issue #49640:
URL: https://github.com/apache/arrow/issues/49640#issuecomment-4177833293

   > It seems that we pick the flags from `/proc/cpuinfo` and that contains the host CPU flags not the target
   
   Ho wow, I though we were using CPUID, similar to on Windows.
   Fortunately there is ongoing work to fully refactor the whole CpuInfo class.
   
   > it seems to me that is an issue under qemu.
   
   I second that diagnosis
   
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] pyarrow._compute Contains AVX Instructions During Import [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #49640:
URL: https://github.com/apache/arrow/issues/49640#issuecomment-4176272079

   I've been able to replicate and got to this specific function registration `arrow::compute::internal::AddSumAvx2AggKernels(arrow::compute::ScalarAggregateFunction*)` from the backtrace:
   ```
   Reading /home/raulcd/code/test-avx/lib/python3.13/site-packages/pyarrow/_compute.cpython-313-x86_64-linux-gnu.so from remote target...                                                                                
   [New Thread 70315.70417]
   ...
   
   Thread 1 received signal SIGILL, Illegal instruction.
   0x00007fda73a3cb42 in arrow::compute::internal::AddSumAvx2AggKernels(arrow::compute::ScalarAggregateFunction*) ()
      from target:/home/raulcd/code/test-avx/lib/python3.13/site-packages/pyarrow/libarrow_compute.so.2300
   ```
   The guard seems correct to me:
   https://github.com/apache/arrow/blob/82847d8f6dae87e709d284b4b871da1424e93a25/cpp/src/arrow/compute/kernels/aggregate_basic.cc#L1065-L1069
   
   It seems that we pick the flags from `/proc/cpuinfo` and that contains the host CPU flags not the target so it seems to me that is an issue under qemu.
   
   cc @AntoinePrv @pitrou what are your thoughts?
   
   BTW `ARROW_USER_SIMD_LEVEL` works for me:
   ```
   $ qemu-x86_64 -cpu Westmere -L / -E ARROW_USER_SIMD_LEVEL=SSE4_2 $(which python) -c 'import pyarrow._compute; import pyarrow; print(pyarrow.__version__)'
   23.0.1
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python][C++] pyarrow._compute Contains AVX Instructions During Import [arrow]

Posted by "Alvaro-Kothe (via GitHub)" <gi...@apache.org>.
Alvaro-Kothe commented on issue #49640:
URL: https://github.com/apache/arrow/issues/49640#issuecomment-4700469980

   Just built arrow from source on ca47cd10b651a8ff5fe44fbb23f7e01dc982a57d and I can confirm that it solved the problem. I also managed to reproduce the issue on its parent.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python][C++] pyarrow._compute Contains AVX Instructions During Import [arrow]

Posted by "Alvaro-Kothe (via GitHub)" <gi...@apache.org>.
Alvaro-Kothe commented on issue #49640:
URL: https://github.com/apache/arrow/issues/49640#issuecomment-4176499931

   > BTW ARROW_USER_SIMD_LEVEL works for me
   
   Indeed, it works. I had a typo when I first tried it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python][C++] pyarrow._compute Contains AVX Instructions During Import [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #49640:
URL: https://github.com/apache/arrow/issues/49640#issuecomment-4692936106

   Ok, let's close. @Alvaro-Kothe if you can still reproduce with PyArrow 25.0.0 (which should be out somewhere in July), feel free to ping here or to open a new issue. Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python][C++] pyarrow._compute Contains AVX Instructions During Import [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #49640:
URL: https://github.com/apache/arrow/issues/49640#issuecomment-4682787781

   @AntoinePrv Do you think this issue can be closed now that use xsimd for CPU detection?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python][C++] pyarrow._compute Contains AVX Instructions During Import [arrow]

Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on issue #49640:
URL: https://github.com/apache/arrow/issues/49640#issuecomment-4691493303

   It should be yes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org