You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "pitrou (via GitHub)" <gi...@apache.org> on 2025/07/17 08:44:37 UTC
[I] [Python] test_cuda_numba_interop fails with numba-cuda [arrow]
pitrou opened a new issue, #47128:
URL: https://github.com/apache/arrow/issues/47128
### Describe the bug, including details regarding any error messages, version, and platform.
Integrated CUDA support in Numba is [deprecated](https://numba.readthedocs.io/en/stable/cuda/overview.html#cuda-deprecation-status) in favor of the external [Numba-CUDA](https://nvidia.github.io/numba-cuda/) module, maintained by NVidia.
However, when running our [Numba CUDA interop tests](https://github.com/apache/arrow/blob/main/python/pyarrow/tests/test_cuda_numba_interop.py) with Numba-CUDA installed, I get the following failures on all tests:
```
File "/home/antoine/arrow/dev/python/pyarrow/tests/test_cuda_numba_interop.py", line 41, in setup_module
ctx2 = cuda.Context.from_numba(nb_ctx2)
File "pyarrow/_cuda.pyx", line 84, in pyarrow._cuda.Context.from_numba
handle=context.handle.value)
AttributeError: 'cuda.bindings.driver.CUcontext' object has no attribute 'value'
```
### Component(s)
GPU, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] test_cuda_numba_interop fails with numba-cuda [arrow]
Posted by "gmarkall (via GitHub)" <gi...@apache.org>.
gmarkall commented on issue #47128:
URL: https://github.com/apache/arrow/issues/47128#issuecomment-3083261533
I think this might be a bug in Numba-CUDA that is fixed in `main` but not yet in a release - let me check and get back to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] test_cuda_numba_interop fails with numba-cuda [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #47128:
URL: https://github.com/apache/arrow/issues/47128#issuecomment-3190282200
Issue resolved by pull request 47150
https://github.com/apache/arrow/pull/47150
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] test_cuda_numba_interop fails with numba-cuda [arrow]
Posted by "gmarkall (via GitHub)" <gi...@apache.org>.
gmarkall commented on issue #47128:
URL: https://github.com/apache/arrow/issues/47128#issuecomment-3085641808
Most of the test failures are resolved by https://github.com/NVIDIA/numba-cuda/pull/328 and https://github.com/NVIDIA/numba-cuda/pull/296.
There is one remaining issue which is not determined by whether Numba or Numba-CUDA is in use, but instead whether the Numba ctypes driver bindings or the NVIDIA CUDA Python driver bindings are in use - ctypes is the default in Numba, and the NVIDIA bindings are the default in Numba-CUDA. I'll look into a fix for this remaining issue tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] test_cuda_numba_interop fails with numba-cuda [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #47128:
URL: https://github.com/apache/arrow/issues/47128#issuecomment-3089151589
No, I'll let you make a PR. This is not urgent since we have just released 21.0.0 and the next release will be in 3 months.
It would be nice if the code remains compatible with the old CUDA support shipped with Numba. I assume that shouldn't be too hard?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] test_cuda_numba_interop fails with numba-cuda [arrow]
Posted by "gmarkall (via GitHub)" <gi...@apache.org>.
gmarkall commented on issue #47128:
URL: https://github.com/apache/arrow/issues/47128#issuecomment-3089306266
> It would be nice if the code remains compatible with the old CUDA support shipped with Numba. I assume that shouldn't be too hard?
Yes - I consider it a hard requirement that Numba and Numba-CUDA are compatible for the CUDA target (notwithstanding new additions to Numba-CUDA that aren't present in upstream Numba). Any discrepancy or breaking of existing code is a bug, not a feature 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] test_cuda_numba_interop fails with numba-cuda [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #47128:
URL: https://github.com/apache/arrow/issues/47128#issuecomment-3083194499
@gmarkall Perhaps this is something that you or one of your colleagues might be interested in submitting a fix for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] test_cuda_numba_interop fails with numba-cuda [arrow]
Posted by "gmarkall (via GitHub)" <gi...@apache.org>.
gmarkall commented on issue #47128:
URL: https://github.com/apache/arrow/issues/47128#issuecomment-3096411260
> No, I'll let you make a PR.
PR in #47150.
> It would be nice if the code remains compatible with the old CUDA support shipped with Numba. I assume that shouldn't be too hard?
I wasn't quite sure how to get the CI config to test with both, but assuming that can be made to work then it should be possible to ensure correctness with both upstream Numba's CUDA target and NVIDIA's numba-cuda.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] test_cuda_numba_interop fails with numba-cuda [arrow]
Posted by "gmarkall (via GitHub)" <gi...@apache.org>.
gmarkall commented on issue #47128:
URL: https://github.com/apache/arrow/issues/47128#issuecomment-3089309472
For anyone using the existing Arrow release with Numba-CUDA, setting `NUMBA_CUDA_USE_NVIDIA_BINDING=0` will override its default setting to use the NVIDIA CUDA Python bindings, and therefore allow it to work without any of the PyArrow or unreleased Numba-CUDA changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] test_cuda_numba_interop fails with numba-cuda [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #47128:
URL: https://github.com/apache/arrow/issues/47128#issuecomment-3083213072
Also, ideally this should also be exercised on our CI. We do have a [CUDA Python CI job](https://github.com/ursacomputing/crossbow/actions/runs/16333402460/job/46140737513), but it [doesn't install Numba-CUDA](https://github.com/apache/arrow/blob/aabd3db3cab180ed55e61df0cc4ee8714f6fd87a/ci/scripts/install_numba.sh#L37-L43), so the deprecated CUDA support is used instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] test_cuda_numba_interop fails with numba-cuda [arrow]
Posted by "gmarkall (via GitHub)" <gi...@apache.org>.
gmarkall commented on issue #47128:
URL: https://github.com/apache/arrow/issues/47128#issuecomment-3085663610
Oops, I forgot to write up the remaining issue - it is:
```
______________________________________ test_numba_memalloc[float32-numba.cuda] _______________________________________
c = 1, dtype = dtype('float32')
@pytest.mark.parametrize("c", range(len(context_choice_ids)),
ids=context_choice_ids)
@pytest.mark.parametrize("dtype", dtypes, ids=dtypes)
def test_numba_memalloc(c, dtype):
ctx, nb_ctx = context_choices[c]
dtype = np.dtype(dtype)
# Allocate memory using numba context
# Warning: this will not be reflected in pyarrow context manager
# (e.g bytes_allocated does not change)
size = 10
mem = nb_ctx.memalloc(size * dtype.itemsize)
darr = DeviceNDArray((size,), (dtype.itemsize,), dtype, gpu_data=mem)
darr[:5] = 99
darr[5:] = 88
np.testing.assert_equal(darr.copy_to_host()[:5], 99)
np.testing.assert_equal(darr.copy_to_host()[5:], 88)
# wrap numba allocated memory with CudaBuffer
> cbuf = cuda.CudaBuffer.from_numba(mem)
pyarrow/tests/test_cuda_numba_interop.py:178:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> if mem.device_pointer.value is None and mem.size==0:
E AttributeError: 'cuda.bindings.driver.CUdeviceptr' object has no attribute 'value'
```
`memalloc` returns `c_void_p` objects with the ctypes bindings, and `CUdeviceptr` objects with the NVIDIA bindings. This is obviously an unpleasant inconsistency, but I've not yet thought of a straightforward fix because other libraries had adapted to this inconsistency (e.g. https://github.com/rapidsai/rmm), so just straightforwardly making it consistent will break them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] test_cuda_numba_interop fails with numba-cuda [arrow]
Posted by "gmarkall (via GitHub)" <gi...@apache.org>.
gmarkall commented on issue #47128:
URL: https://github.com/apache/arrow/issues/47128#issuecomment-3089063774
With this change in Arrow:
```diff
diff --git a/python/pyarrow/_cuda.pyx b/python/pyarrow/_cuda.pyx
index 340f7e35c7..96e5c13949 100644
--- a/python/pyarrow/_cuda.pyx
+++ b/python/pyarrow/_cuda.pyx
@@ -451,9 +451,9 @@ cdef class CudaBuffer(Buffer):
Device buffer as a view of numba MemoryPointer.
"""
ctx = Context.from_numba(mem.context)
- if mem.device_pointer.value is None and mem.size==0:
+ if mem.device_pointer_value is None and mem.size==0:
return ctx.new_buffer(0)
- return ctx.foreign_buffer(mem.device_pointer.value, mem.size, base=mem)
+ return ctx.foreign_buffer(mem.device_pointer_value, mem.size, base=mem)
def to_numba(self):
"""Return numba memory pointer of CudaBuffer instance.
```
and the Numba-CUDA PR https://github.com/NVIDIA/numba-cuda/pull/328 on top of Numba-CUDA `main`, all tests pass:
```
0 gmarkall@housel 11:32:46 (main) ~/numbadev/arrow/python
$ python -m pytest pyarrow/tests/test_cuda_numba_interop.py
============================================ test session starts =============================================
platform linux -- Python 3.10.18, pytest-8.4.1, pluggy-1.6.0
rootdir: /home/gmarkall/numbadev/arrow/python
configfile: setup.cfg
plugins: hypothesis-6.135.32
collected 50 items
pyarrow/tests/test_cuda_numba_interop.py .................................................. [100%]
====================================== 50 passed, 18 warnings in 2.48s =======================================
0 gmarkall@housel 11:32:50 (main) ~/numbadev/arrow/python
$ pip list
...
numba-cuda 0.16.0 /home/gmarkall/numbadev/numba-cuda
...
```
Next I'll aim to:
* Get a Numba-CUDA release out with the relevant fixes, and
* Make a PR to Arrow with the above change.
The change above swaps the use of `device_pointer.value` with `device_pointer_value` because `device_pointer_value` always returns `None` or an integer regardless of whether the `device_pointer` is a `c_void_p` or `CUdeviceptr`. Feel free to take this change and merge it before I get chance to make a PR, if you would prefer :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] test_cuda_numba_interop fails with numba-cuda [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou closed issue #47128: [Python] test_cuda_numba_interop fails with numba-cuda
URL: https://github.com/apache/arrow/issues/47128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org