You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "AlenkaF (via GitHub)" <gi...@apache.org> on 2024/05/30 12:33:10 UTC

[PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

AlenkaF opened a new pull request, #41889:
URL: https://github.com/apache/arrow/pull/41889

   ### Rationale for this change
   
   `hex()` and `__getitem__` currently segfault if the data is not located on the CPU.
   
   ### What changes are included in this PR?
   
   This PR adds a check and returns `NotImplementedError` if the data is not on CPU. 
   
   ### Are these changes tested?
   
   Yes
   
   ### Are there any user-facing changes?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1632751914


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    buf = pa.py_buffer(b'testing')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, buf])
+    buf_on_gpu = arr.buffers()[1]
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+
+    buf = pa.py_buffer(b'sting')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(5), 1, [None, buf])
+    buf_on_gpu_expected = arr.buffers()[1]
+
+    assert buf_on_gpu_sliced.equals(buf_on_gpu_expected)
+    assert buf.equals(buf_on_gpu_expected)

Review Comment:
   > I am not sure if I can check that the `slice` method works correctly as I can't use the `to_pybytes()`
   
   But to_pybytes should work on a CudaBuffer, I think?
   
   > `equals` doesn't seem to work.
   
   I also don't see this explicitly implemented for CudaBuffer on the C++ side (so if that uses the base Buffer::Equals implementation, this is probably expected to crash because it is comparing memory as if it is on the CPU). So let's protect from calling that for now on non-CPU, but should maybe open an issue about that whether it is worth implementing CudaBuffer::Equals.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2163382637

   @github-actions crossbow submit test-cuda-python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1636340616


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,62 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.array([b'testing'])
+    cuda_buf = ctx.buffer_from_data(arr)
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+    assert buf_on_gpu.is_mutable
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=7 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+    cuda_sliced = cuda.CudaBuffer.from_buffer(buf_on_gpu_sliced)
+    assert cuda_sliced.to_pybytes() == b'sting'
+
+    buf_on_gpu_sliced = buf_on_gpu[slice(2, 4, 1)]

Review Comment:
   ```suggestion
       buf_on_gpu_sliced = buf_on_gpu[2:4:1]
   ```
   
   (that's doing exactly the same, but a bit more typical way of spelling it)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1629849391


##########
python/pyarrow/io.pxi:
##########
@@ -1311,7 +1311,10 @@ cdef class Buffer(_Weakrefable):
         -------
         : bytes
         """
-        return self.buffer.get().ToHexString()
+        if self.is_cpu:
+            return self.buffer.get().ToHexString()
+        else:
+            raise NotImplementedError("Implemented only for data on CPU device")

Review Comment:
   ```suggestion
           if not self.is_cpu:
               raise NotImplementedError("Implemented only for data on CPU device")
           return self.buffer.get().ToHexString()            
   ```
   
   (in this case your code is also perfectly fine, but I think I have a slight preference over always using the same pattern of checking if not CPU. Although it is only two lines and we only need to do it in three places for now, it could also be factored out in a helper method like `self._assert_cpu()`)



##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    buf = pa.py_buffer(b'testing')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, buf])
+    buf_on_gpu = arr.buffers()[1]
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+
+    buf = pa.py_buffer(b'sting')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(5), 1, [None, buf])
+    buf_on_gpu_expected = arr.buffers()[1]
+
+    assert buf_on_gpu_sliced.equals(buf_on_gpu_expected)
+    assert buf.equals(buf_on_gpu_expected)
+    assert buf_on_gpu_sliced.to_pybytes() == buf_on_gpu_expected.to_pybytes()

Review Comment:
   Could you also check `buf_on_gpu.to_pybytes()`  with a hardcoded bytes result? Because right now if both left and right return the same but wrong result, this will pass ..
   
   (looking at `to_pybytes` implementation, I would actually expect this to not work properly)



##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)

Review Comment:
   Similarly can you again add a version testing this with `cuda_buf` as well?



##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    buf = pa.py_buffer(b'testing')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, buf])
+    buf_on_gpu = arr.buffers()[1]
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+
+    buf = pa.py_buffer(b'sting')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(5), 1, [None, buf])
+    buf_on_gpu_expected = arr.buffers()[1]
+
+    assert buf_on_gpu_sliced.equals(buf_on_gpu_expected)
+    assert buf.equals(buf_on_gpu_expected)

Review Comment:
   I find it surprising that this passes for checking equality with CPU and non-CPU buffer ..
   
   But can you also check `buf_on_gpu.equals(cuda_buf)` above? (so combining a Buffer and CudaBuffer object) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1625665673


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,48 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer():
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    import numpy as np

Review Comment:
   ```suggestion
   ```
   
   (already imported at the top)



##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,48 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer():
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    import numpy as np
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    buf = pa.py_buffer(b'testing')
+    arr_offsets = pa.array([0, 7], type=pa.int32())
+    offsets_buf = arr_offsets.buffers()[1]

Review Comment:
   ```suggestion
       offsets_buf = pa.py_buffer(np.array([0, 7], np.int32))
   ```
   
   (maybe a bit simpler)



##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,48 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer():
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    import numpy as np
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()

Review Comment:
   Can you test this with `cuda_buf.hex()` as well? 
   
   (Because the CudaBuffer class is subclassing, it therefore has this method available as well from the parent class, and it doesn't seem to be tested in test_cuda.py right now)



##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,48 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer():
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    import numpy as np
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    buf = pa.py_buffer(b'testing')
+    arr_offsets = pa.array([0, 7], type=pa.int32())
+    offsets_buf = arr_offsets.buffers()[1]

Review Comment:
   Or even simpler would be to use a FixedSizeBinaryArray, then you don't need offsets to start with



##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,48 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer():
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    import numpy as np
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]

Review Comment:
   Same here for `cuda_buf` (although it _might_ work in that case, as I see it has a `getitem` defined)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2168233933

   After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit f43cf0d6146068b8a1a1f61be248dc7fa6fc6591.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/26238785485) has more details. It also includes information about 7 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1636435281


##########
python/pyarrow/tests/test_cuda.py:
##########
@@ -313,7 +313,8 @@ def test_CudaBuffer(size):
         assert cbuf[s].to_pybytes() == arr[s].tobytes()
 
     sbuf = cbuf.slice(size//4, size//2)
-    assert sbuf.parent == cbuf
+    cbuf_parent = cuda.CudaBuffer.from_buffer(sbuf.parent)
+    assert cbuf_parent.to_pybytes() == cbuf.to_pybytes()

Review Comment:
   Oh, I think I know.
   
   If I compared buffers coming out from the same memory location the `equals` method worked (for example sliced pyarrow Buffer on gpu and sliced CUDA buffer from which the pyarrow Buffer was created, or the example in the cuda test). But if I compared buffers on gpu but on a different memory location, it failed.
   
   Not sure it makes sense - will add a repr (working on it).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2162845537

   Revision: e254bc210b4d7e9e225aaf4c4bd7594e4f013eac
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-76e746192a](https://github.com/ursacomputing/crossbow/branches/all?query=actions-76e746192a)
   
   |Task|Status|
   |----|------|
   |test-cuda-python|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-76e746192a-github-test-cuda-python)](https://github.com/ursacomputing/crossbow/actions/runs/9482251745/job/26126742158)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2162841606

   @github-actions crossbow submit test-cuda-python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1629952428


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    buf = pa.py_buffer(b'testing')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, buf])
+    buf_on_gpu = arr.buffers()[1]
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+
+    buf = pa.py_buffer(b'sting')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(5), 1, [None, buf])
+    buf_on_gpu_expected = arr.buffers()[1]
+
+    assert buf_on_gpu_sliced.equals(buf_on_gpu_expected)
+    assert buf.equals(buf_on_gpu_expected)
+    assert buf_on_gpu_sliced.to_pybytes() == buf_on_gpu_expected.to_pybytes()

Review Comment:
   Actually I think it was failing with integer data. Will check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1620807811


##########
python/pyarrow/io.pxi:
##########
@@ -1337,6 +1340,9 @@ cdef class Buffer(_Weakrefable):
             return pyarrow_wrap_buffer(parent_buf)
 
     def __getitem__(self, key):
+        if (not self.is_cpu):

Review Comment:
   FWIW You don't need the brackets here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1629981539


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    buf = pa.py_buffer(b'testing')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, buf])
+    buf_on_gpu = arr.buffers()[1]
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+
+    buf = pa.py_buffer(b'sting')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(5), 1, [None, buf])
+    buf_on_gpu_expected = arr.buffers()[1]
+
+    assert buf_on_gpu_sliced.equals(buf_on_gpu_expected)
+    assert buf.equals(buf_on_gpu_expected)

Review Comment:
   Checking equation between `CudaBuffer` and `Buffer` on gpu work. But I get a segfault if checking equation between two `Buffers` on gpu (`buf_on_gpu_sliced` and `buf_on_gpu_expected`): https://github.com/apache/arrow/pull/41889/commits/958af940b3020f5656e86a08ee9f5d47d9857660



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1638112968


##########
python/pyarrow/io.pxi:
##########
@@ -1424,6 +1430,18 @@ cdef class Buffer(_Weakrefable):
         are_equal : bool
             True if buffer contents and size are equal
         """
+        if self.device != other.device:
+            raise ValueError(
+                "Device on which the data resides differs between buffers: "
+                f"{self.device.type_name} and {other.device.type_name}."
+            )
+        if not self.is_cpu and not other.is_cpu:

Review Comment:
   I think the current error message could actually be clearer? i.e. if you are comparing a buffer on two different devices it will explicitly say that that is not supported because they are on a different device



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1636121244


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,64 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+    assert buf_on_gpu.is_mutable
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.equals(cuda_buf)
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.equals(buf_on_gpu)
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]

Review Comment:
   I added a test for `buf_on_gpu` as the cuda buffers accessing with index and slice is tested in _test_CudaBuffer_: https://github.com/apache/arrow/pull/41889/commits/a4d409a5008c7266f8d18e5bd3e42e3ce95e222b
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1632656493


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    buf = pa.py_buffer(b'testing')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, buf])
+    buf_on_gpu = arr.buffers()[1]
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+
+    buf = pa.py_buffer(b'sting')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(5), 1, [None, buf])
+    buf_on_gpu_expected = arr.buffers()[1]
+
+    assert buf_on_gpu_sliced.equals(buf_on_gpu_expected)
+    assert buf.equals(buf_on_gpu_expected)

Review Comment:
   `equals` doesn't seem to work. Example:
   
   ```python
   >>> arr = np.array([b'testing'])
   >>> cuda_buf = ctx.buffer_from_data(arr)
   >>> arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, cuda_buf])
   >>> buf_on_gpu = arr.buffers()[1]
   
   >>> arr2 = np.array([b'testing'])
   >>> cuda_buf2 = ctx.buffer_from_data(arr2)
   >>> arr2 = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, cuda_buf2])
   >>> buf_on_gpu2 = arr2.buffers()[1]
   
   >>> buf_on_gpu.equals(buf_on_gpu2)
   Segmentation fault
   ```
   
   I am not sure if I can check that the `slice` method works correctly as I can't use the `to_pybytes()` or `equals()` methods. Comparing to cuda buffer also segfauls in this case:
   
   ```python
   >>> arr = np.array([b'testing'])
   >>> cuda_buf = ctx.buffer_from_data(arr)
   >>> arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, cuda_buf])
   >>> buf_on_gpu = arr.buffers()[1]
   
   >>> buf_on_gpu.equals(cuda_buf)
   True
   >>> buf_on_gpu_sliced = buf_on_gpu.slice(2)
   
   >>> arr_expected = np.array([b'sting'])
   >>> cuda_buf_expected = ctx.buffer_from_data(arr_expected)
   
   >>> buf_on_gpu_sliced.equals(cuda_buf_expected)
   Segmentation fault
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2152495640

   > Can you try with specifying `protocol=4` passed to `dumps()` ?
   
   Passes! I get assertion error for:
   
   ```python
   >       assert memoryview(result) == memoryview(buf_on_gpu)
   
   pyarrow/tests/test_io.py:693: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
   
   >   assert buffer.len == 0
   E   AssertionError
   
   pyarrow/io.pxi:1444: AssertionError
   ```
   
   and
   
   ```python
           result = pickle_module.loads(pickle_module.dumps(buf_on_gpu, protocol=4))
           assert len(result) == len(buf_on_gpu)
           # assert memoryview(result) == memoryview(buf_on_gpu)
   >       assert result.to_pybytes() == buf_on_gpu.to_pybytes()
   E       AssertionError: assert b'\x10R\x9c\x...\xc5U\x00\x00' == b'\x00\x00\x0...0\x00\x00\x00'
   E         
   E         At index 0 diff: b'\x10' != b'\x00'
   E         Use -v to get more diff
   
   pyarrow/tests/test_io.py:694: AssertionError
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1629937919


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    buf = pa.py_buffer(b'testing')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, buf])
+    buf_on_gpu = arr.buffers()[1]
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+
+    buf = pa.py_buffer(b'sting')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(5), 1, [None, buf])
+    buf_on_gpu_expected = arr.buffers()[1]
+
+    assert buf_on_gpu_sliced.equals(buf_on_gpu_expected)
+    assert buf.equals(buf_on_gpu_expected)
+    assert buf_on_gpu_sliced.to_pybytes() == buf_on_gpu_expected.to_pybytes()

Review Comment:
   Ah yes, it was failing and I thought it was something I did wrong.
   Will look into it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1636382975


##########
python/pyarrow/tests/test_cuda.py:
##########
@@ -313,7 +313,8 @@ def test_CudaBuffer(size):
         assert cbuf[s].to_pybytes() == arr[s].tobytes()
 
     sbuf = cbuf.slice(size//4, size//2)
-    assert sbuf.parent == cbuf
+    cbuf_parent = cuda.CudaBuffer.from_buffer(sbuf.parent)
+    assert cbuf_parent.to_pybytes() == cbuf.to_pybytes()

Review Comment:
   What's the reason that this worked before? (it's a bit strange we would have to change anything about the tests here)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1638043444


##########
python/pyarrow/io.pxi:
##########
@@ -1424,6 +1430,18 @@ cdef class Buffer(_Weakrefable):
         are_equal : bool
             True if buffer contents and size are equal
         """
+        if self.device != other.device:
+            raise ValueError(
+                "Device on which the data resides differs between buffers: "
+                f"{self.device.type_name} and {other.device.type_name}."
+            )
+        if not self.is_cpu and not other.is_cpu:

Review Comment:
   ```suggestion
           if not self.is_cpu:
   ```
   
   (because you now already checked the devices are equal, just checking for `self.is_cpu` should be sufficient)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2162883493

   Ah, one commit is left and didn't get pushed as I disconnected the VPN 😊 Will push now and re-run the cuda test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1632828921


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    buf = pa.py_buffer(b'testing')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, buf])
+    buf_on_gpu = arr.buffers()[1]
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+
+    buf = pa.py_buffer(b'sting')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(5), 1, [None, buf])
+    buf_on_gpu_expected = arr.buffers()[1]
+
+    assert buf_on_gpu_sliced.equals(buf_on_gpu_expected)
+    assert buf.equals(buf_on_gpu_expected)

Review Comment:
   Ah yes, but so you can always convert a Buffer with Cuda device to a CudaBuffer with `CudaBuffer.from_buffer(bug_on_gpu)`, and then you can use that method. 
   
   Now, CudaBuffer.to_pybytes also just copies to the data to the host, so we should probably add similar logic for `Buffer.to_pybytes` (but this first requires to expose the copy methods to different device, so that's for a follow-up)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2152528984

   Anyway, the reason this fails is because I think pickling is not expected to work with non-CPU data. Even if it does not segfault, we should not allow calling it if it simply gives garbage results (it seems all null in the second case). I would add a similar error to `__reduce_ex__` to avoid getting those confusing errors or wrong results.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2152960331

   @github-actions crossbow submit test-cuda-python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2152430000

   > Some more suggestions for the tests:
   > 
   > * test the repr
   > * does the `equals()` method work? (both for cuda == cuda or cuda == cpu)
   
   I have added all together with the other suggestions.
   
   > * pickling might also segfault?
   
   It doesn't look it segfaults. I get this error though:
   
   ```python
   >       result = pickle_module.loads(pickle_module.dumps(buf_on_gpu))
   
   pyarrow/tests/test_io.py:692: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
   ../../envs/arrow-dev/lib/python3.11/site-packages/cloudpickle/cloudpickle.py:1479: in dumps
       cp.dump(obj)
   ../../envs/arrow-dev/lib/python3.11/site-packages/cloudpickle/cloudpickle.py:1245: in dump
       return super().dump(obj)
   pyarrow/io.pxi:1413: in pyarrow.lib.Buffer.__reduce_ex__
       bufobj = pickle.PickleBuffer(self)
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
   
   >   assert buffer.len == 0
   E   AssertionError
   
   pyarrow/io.pxi:1444: AssertionError
   ```
   
   Haven't looked into it yet. Will do.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2152964695

   Revision: 3f56e121dc46af4d7a95c7154224d18f1a0a535f
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-b43dfa1819](https://github.com/ursacomputing/crossbow/branches/all?query=actions-b43dfa1819)
   
   |Task|Status|
   |----|------|
   |test-cuda-python|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b43dfa1819-github-test-cuda-python)](https://github.com/ursacomputing/crossbow/actions/runs/9404673437/job/25904094964)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2158070958

   @github-actions crossbow submit test-cuda-python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1632810044


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    buf = pa.py_buffer(b'testing')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, buf])
+    buf_on_gpu = arr.buffers()[1]
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+
+    buf = pa.py_buffer(b'sting')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(5), 1, [None, buf])
+    buf_on_gpu_expected = arr.buffers()[1]
+
+    assert buf_on_gpu_sliced.equals(buf_on_gpu_expected)
+    assert buf.equals(buf_on_gpu_expected)

Review Comment:
   > But to_pybytes should work on a CudaBuffer, I think?
   
   Yes, `to_pybytes` works correctly for CUDA buffer. But we are looking at the arrow buffer constructed from a CUDA one, right? So I can not check slicing for arrow buffer on gpu.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1630758114


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    buf = pa.py_buffer(b'testing')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, buf])
+    buf_on_gpu = arr.buffers()[1]
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+
+    buf = pa.py_buffer(b'sting')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(5), 1, [None, buf])
+    buf_on_gpu_expected = arr.buffers()[1]
+
+    assert buf_on_gpu_sliced.equals(buf_on_gpu_expected)
+    assert buf.equals(buf_on_gpu_expected)
+    assert buf_on_gpu_sliced.to_pybytes() == buf_on_gpu_expected.to_pybytes()

Review Comment:
   If you look at the implementation of `to_pybytes`, this is expected:
   
   https://github.com/apache/arrow/blob/01d2fa0d461869a07b2ffeee517beb8116bd0ce2/python/pyarrow/io.pxi#L1451-L1457
   
   This gets the pointer to the buffer (`data()`) and passes that to a Python function. But so Python will just assume this is a pointer to some memory on the CPU, and thus if it gets a pointer to GPU memory and interpret that as CPU it's just looking at some random memory, giving garbage.
   
   So again like the other methods, we can add an error in case of non-CPU data.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2162244151

   The reason it works for CudaBuffer is because they override the `cdef getitem` method that is called from `__getitem__`. So I think the better way to solve it is to not call `self._assert_cpu()` inside `__getitem__` but only in `getitem` (`__getitem__` actually also works in case of slicing for any device, since `slice()` works, so that is also something we should enable)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1636718079


##########
python/pyarrow/tests/test_cuda.py:
##########
@@ -313,7 +313,8 @@ def test_CudaBuffer(size):
         assert cbuf[s].to_pybytes() == arr[s].tobytes()
 
     sbuf = cbuf.slice(size//4, size//2)
-    assert sbuf.parent == cbuf
+    cbuf_parent = cuda.CudaBuffer.from_buffer(sbuf.parent)
+    assert cbuf_parent.to_pybytes() == cbuf.to_pybytes()

Review Comment:
   Looking at `Buffer::Equals`, as you suggested, I understand the behaviour I described in the last comment of this thread. If the addresses of both buffers are equal then `Equals` is true even if they live on GPU. But if the address is different the method uses `memcmp` which will fail for data that is not on CPU.
   
   I added a different check for equals method that checks addresses of buffers that do not live on cpu and raises an error only if they are not the same: https://github.com/apache/arrow/pull/41889/commits/1268c1be2bb863ff300e70e5fbd73047e4e9da5c.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche merged PR #41889:
URL: https://github.com/apache/arrow/pull/41889


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1629965069


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    buf = pa.py_buffer(b'testing')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, buf])
+    buf_on_gpu = arr.buffers()[1]
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+
+    buf = pa.py_buffer(b'sting')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(5), 1, [None, buf])
+    buf_on_gpu_expected = arr.buffers()[1]
+
+    assert buf_on_gpu_sliced.equals(buf_on_gpu_expected)
+    assert buf.equals(buf_on_gpu_expected)
+    assert buf_on_gpu_sliced.to_pybytes() == buf_on_gpu_expected.to_pybytes()

Review Comment:
   Yes, the case with integer data is wrong:
   
   ```python
   >>> from pyarrow import cuda
   >>> ctx = cuda.Context(0)
   >>> import numpy as np
   >>> import pyarrow as pa
   
   >>> arr = np.arange(4, dtype=np.int32)
   >>> cuda_buf = ctx.buffer_from_data(arr)
   
   >>> arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
   >>> buf_on_gpu = arr.buffers()[1]
   
   >>> buf_on_gpu.to_pybytes()
   b'\xe4\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
   >>> cuda_buf.to_pybytes()
   b'\x00\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00'
   ```
   
   and I think the binary would be also - I just noticed I am starting from a `pyarrow.Buffer `in this case, not CUDA Buffer! Need to correct that.
   
   Will also need to look into why the `to_pybytes` is wrong and is it only for non-CPU data? I would guess it worked correctly till now for CPU data?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1632887819


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    buf = pa.py_buffer(b'testing')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, buf])
+    buf_on_gpu = arr.buffers()[1]
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+
+    buf = pa.py_buffer(b'sting')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(5), 1, [None, buf])
+    buf_on_gpu_expected = arr.buffers()[1]
+
+    assert buf_on_gpu_sliced.equals(buf_on_gpu_expected)
+    assert buf.equals(buf_on_gpu_expected)

Review Comment:
   Aha, didn't think of going back to CUDA buffer. Will do 👍 
   
   > Now, CudaBuffer.to_pybytes also just copies to the data to the host, so we should probably add similar logic for `Buffer.to_pybytes` (but this first requires to expose the copy methods to different device, so that's for a follow-up)
   
   Yes, that would be great!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2164702541

   Also a linting error to fix


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2165274831

   @github-actions crossbow submit test-cuda-python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2162887464

   @github-actions crossbow submit test-cuda-python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2163387759

   Revision: e0ad972acf29404dd7d6175c7ec8deb579398494
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-5e22d7f919](https://github.com/ursacomputing/crossbow/branches/all?query=actions-5e22d7f919)
   
   |Task|Status|
   |----|------|
   |test-cuda-python|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-5e22d7f919-github-test-cuda-python)](https://github.com/ursacomputing/crossbow/actions/runs/9485733097/job/26138385741)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1636382975


##########
python/pyarrow/tests/test_cuda.py:
##########
@@ -313,7 +313,8 @@ def test_CudaBuffer(size):
         assert cbuf[s].to_pybytes() == arr[s].tobytes()
 
     sbuf = cbuf.slice(size//4, size//2)
-    assert sbuf.parent == cbuf
+    cbuf_parent = cuda.CudaBuffer.from_buffer(sbuf.parent)
+    assert cbuf_parent.to_pybytes() == cbuf.to_pybytes()

Review Comment:
   What's the reason that this worked before?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1638114990


##########
python/pyarrow/io.pxi:
##########
@@ -1424,6 +1430,18 @@ cdef class Buffer(_Weakrefable):
         are_equal : bool
             True if buffer contents and size are equal
         """
+        if self.device != other.device:
+            raise ValueError(
+                "Device on which the data resides differs between buffers: "
+                f"{self.device.type_name} and {other.device.type_name}."
+            )
+        if not self.is_cpu and not other.is_cpu:

Review Comment:
   Ok, great! 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2158075210

   Revision: 95633a9c65a48a58e91bd17a35aca1966853100b
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-b2b2369881](https://github.com/ursacomputing/crossbow/branches/all?query=actions-b2b2369881)
   
   |Task|Status|
   |----|------|
   |test-cuda-python|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b2b2369881-github-test-cuda-python)](https://github.com/ursacomputing/crossbow/actions/runs/9447414226/job/26019156530)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1635914530


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,64 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+    assert buf_on_gpu.is_mutable
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.equals(cuda_buf)
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.equals(buf_on_gpu)
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]

Review Comment:
   For getitem, can you then also test this with a slice in addition to integer index, in which case it is expected to work for both buf_on_gpu and cuda_buf



##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,64 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+    assert buf_on_gpu.is_mutable
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.equals(cuda_buf)
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.equals(buf_on_gpu)
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.to_pybytes()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(cuda_buf, protocol=4)
+
+    arr = np.array([b'testing'])
+    cuda_buf = ctx.buffer_from_data(arr)
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]

Review Comment:
   To simplify the test a bit, you could also use this definition of `cuda_buf` and `buf_on_gpu` from the start of the test, and then you don't have to redefine the variables here just for the slicing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1636340616


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,62 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.array([b'testing'])
+    cuda_buf = ctx.buffer_from_data(arr)
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+    assert buf_on_gpu.is_mutable
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=7 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+    cuda_sliced = cuda.CudaBuffer.from_buffer(buf_on_gpu_sliced)
+    assert cuda_sliced.to_pybytes() == b'sting'
+
+    buf_on_gpu_sliced = buf_on_gpu[slice(2, 4, 1)]

Review Comment:
   ```suggestion
       buf_on_gpu_sliced = buf_on_gpu[2:4]
   ```
   
   (that's doing exactly the same, but a bit more typical way of spelling it)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1636423725


##########
python/pyarrow/tests/test_cuda.py:
##########
@@ -313,7 +313,8 @@ def test_CudaBuffer(size):
         assert cbuf[s].to_pybytes() == arr[s].tobytes()
 
     sbuf = cbuf.slice(size//4, size//2)
-    assert sbuf.parent == cbuf
+    cbuf_parent = cuda.CudaBuffer.from_buffer(sbuf.parent)
+    assert cbuf_parent.to_pybytes() == cbuf.to_pybytes()

Review Comment:
   `sbuf.parent` is pyarrow Buffer with data on GPU and so the `equals` method can not work as we have added a check for non-cpu data.
   
   Why it was working before I am not sure - it segfaulted when testing buffers on GPU for tests in _test_io.py_ 🤷‍♀️ 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1637658497


##########
python/pyarrow/tests/test_cuda.py:
##########
@@ -313,7 +313,8 @@ def test_CudaBuffer(size):
         assert cbuf[s].to_pybytes() == arr[s].tobytes()
 
     sbuf = cbuf.slice(size//4, size//2)
-    assert sbuf.parent == cbuf
+    cbuf_parent = cuda.CudaBuffer.from_buffer(sbuf.parent)
+    assert cbuf_parent.to_pybytes() == cbuf.to_pybytes()

Review Comment:
   > I added a different check for equals method that checks addresses of buffers that do not live on cpu and raises an error only if they are not the same: [1268c1b](https://github.com/apache/arrow/commit/1268c1be2bb863ff300e70e5fbd73047e4e9da5c).
   
   Sounds good. Maybe only add a check that the self and other `.device` is equal as well? (because only at that point comparing the address is fine)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2165381436

   Revision: a621676f7d97fa477133df6c82c8ee7ccbec33d3
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-55bfca7575](https://github.com/ursacomputing/crossbow/branches/all?query=actions-55bfca7575)
   
   |Task|Status|
   |----|------|
   |test-cuda-python|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-55bfca7575-github-test-cuda-python)](https://github.com/ursacomputing/crossbow/actions/runs/9498783227/job/26178236185)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2165373127

   @github-actions crossbow submit test-cuda-python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1620856079


##########
python/pyarrow/io.pxi:
##########
@@ -1337,6 +1340,9 @@ cdef class Buffer(_Weakrefable):
             return pyarrow_wrap_buffer(parent_buf)
 
     def __getitem__(self, key):
+        if (not self.is_cpu):

Review Comment:
   Oh, I am getting confused switching between C++ and Python 🤣 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1629970741


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    buf = pa.py_buffer(b'testing')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, buf])
+    buf_on_gpu = arr.buffers()[1]
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+
+    buf = pa.py_buffer(b'sting')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(5), 1, [None, buf])
+    buf_on_gpu_expected = arr.buffers()[1]
+
+    assert buf_on_gpu_sliced.equals(buf_on_gpu_expected)
+    assert buf.equals(buf_on_gpu_expected)
+    assert buf_on_gpu_sliced.to_pybytes() == buf_on_gpu_expected.to_pybytes()

Review Comment:
   Yes, not working:
   
   ```python
   >>> arr = np.array([b'testing'])
   >>> cuda_buf = ctx.buffer_from_data(arr)
   >>> arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, cuda_buf])
   >>> buf_on_gpu = arr.buffers()[1]
   
   >>> cuda_buf.to_pybytes()
   b'testing'
   >>> buf_on_gpu.to_pybytes()
   b'\x15\x00\x00\x00\x00\x00\x00'
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2162891902

   Revision: a1e43491f1630df61ad9ce0567a1e83f3736fc27
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-12b895c3d6](https://github.com/ursacomputing/crossbow/branches/all?query=actions-12b895c3d6)
   
   |Task|Status|
   |----|------|
   |test-cuda-python|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-12b895c3d6-github-test-cuda-python)](https://github.com/ursacomputing/crossbow/actions/runs/9482605684/job/26127844125)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1633080740


##########
python/pyarrow/tests/test_io.py:
##########
@@ -669,6 +669,57 @@ def test_allocate_buffer_resizable():
     assert buf.size == 200
 
 
+def test_non_cpu_buffer(pickle_module):
+    cuda = pytest.importorskip("pyarrow.cuda")
+    ctx = cuda.Context(0)
+
+    arr = np.arange(4, dtype=np.int32)
+    cuda_buf = ctx.buffer_from_data(arr)
+
+    arr = pa.Array.from_buffers(pa.int32(), 4, [None, cuda_buf])
+    buf_on_gpu = arr.buffers()[1]
+
+    assert buf_on_gpu.size == cuda_buf.size
+    assert buf_on_gpu.address == cuda_buf.address
+    assert buf_on_gpu.is_cpu == cuda_buf.is_cpu
+
+    repr1 = "<pyarrow.Buffer address="
+    repr2 = "size=16 is_cpu=False is_mutable=True>"
+    assert repr1 in repr(buf_on_gpu)
+    assert repr2 in repr(buf_on_gpu)
+
+    msg = "Implemented only for data on CPU device"
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu.hex()
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf.hex()
+
+    assert buf_on_gpu.is_mutable
+
+    with pytest.raises(NotImplementedError, match=msg):
+        buf_on_gpu[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        cuda_buf[1]
+
+    with pytest.raises(NotImplementedError, match=msg):
+        pickle_module.dumps(buf_on_gpu, protocol=4)
+
+    buf = pa.py_buffer(b'testing')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(7), 1, [None, buf])
+    buf_on_gpu = arr.buffers()[1]
+    buf_on_gpu_sliced = buf_on_gpu.slice(2)
+
+    buf = pa.py_buffer(b'sting')
+    arr = pa.FixedSizeBinaryArray.from_buffers(pa.binary(5), 1, [None, buf])
+    buf_on_gpu_expected = arr.buffers()[1]
+
+    assert buf_on_gpu_sliced.equals(buf_on_gpu_expected)
+    assert buf.equals(buf_on_gpu_expected)

Review Comment:
   Issue opened: https://github.com/apache/arrow/issues/42053



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2158301367

   Looking at the failing tests in [test-cuda-python](https://github.com/ursacomputing/crossbow/actions/runs/9447414226/job/26019156530):
   
   ```python
   ...
   >           assert cbuf[s].to_pybytes() == arr[s].tobytes()
   
   arrow-dev/lib/python3.8/site-packages/pyarrow/tests/test_cuda.py:313: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   pyarrow/io.pxi:1345: in pyarrow.lib.Buffer.__getitem__
       ???
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   >   ???
   E   NotImplementedError: Implemented only for data on CPU device
   
   pyarrow/io.pxi:1291: NotImplementedError
   ```
   
   `__getitem__`is expected to work and not error with `NotImplementedError` for `CudaBuffer`. This would then be a breaking change. Should I check if it is an instance of `CudaBuffer` and only if it is not raise an error for non-cpu data?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41889:
URL: https://github.com/apache/arrow/pull/41889#discussion_r1638054651


##########
python/pyarrow/io.pxi:
##########
@@ -1424,6 +1430,18 @@ cdef class Buffer(_Weakrefable):
         are_equal : bool
             True if buffer contents and size are equal
         """
+        if self.device != other.device:
+            raise ValueError(
+                "Device on which the data resides differs between buffers: "
+                f"{self.device.type_name} and {other.device.type_name}."
+            )
+        if not self.is_cpu and not other.is_cpu:

Review Comment:
   I rearranged a bit locally - think it makes more sense:
   
   ```python
   if not self.is_cpu or not other.is_cpu:
     if self.device != other.device or self.address != other.address:
         raise NotImplementedError(
             "Implemented only for data on CPU device or data with equal "
             "addresses"
         )
   ```
   
   can push if not too late.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2165279874

   Revision: 5e60233307ac844372af6edaffeea90ef410b1b1
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-d722963ddb](https://github.com/ursacomputing/crossbow/branches/all?query=actions-d722963ddb)
   
   |Task|Status|
   |----|------|
   |test-cuda-python|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-d722963ddb-github-test-cuda-python)](https://github.com/ursacomputing/crossbow/actions/runs/9498181945/job/26176339917)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41662: [Python] Ensure Buffer methods don't crash with non-CPU data [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41889:
URL: https://github.com/apache/arrow/pull/41889#issuecomment-2152475109

   > It doesn't look it segfaults.
   
   Can you try with specifying `protocol=4` passed to `dumps()` ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org