You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2024/05/01 19:39:47 UTC
[PR] feat(python): Implement bitmap unpacking [arrow-nanoarrow]
paleolimbot opened a new pull request, #450:
URL: https://github.com/apache/arrow-nanoarrow/pull/450
In prototyping a real-world use case, I remembered that unpacking bits is exceedingly difficult to get right if you need to support an arbitrary offset/length. The math for this is very fiddly and we spent a few rounds getting it right in the C function `ArrowBitsUnpackInt(8|32)`. This PR makes that available so that we can do things like (1) convert bool arrays to numpy and (2) convert null masks to something that somebody else can work with (e.g., a numpy mask).
This seems to be relatively performant (thanks to @WillAyd's work optimizing this!)
```python
import numpy as np
import nanoarrow as na
import pyarrow as pa
bool_np = np.random.random(int(1e6)) > 0.5
bool_na = na.Array(iter(bool_array), na.bool_())
bool_pa = pa.array(bool_np)
def to_numpy_na(x):
x_view = na.c_array(x).view()
out = np.empty(x_view.length, bool)
x_view.buffer(1).unpack_bits_into(out, x_view.offset, x_view.length)
return out
%timeit to_numpy_na(bool_na)
#> 162 µs ± 812 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
%timeit bool_pa.to_numpy(False)
#> 609 µs ± 833 ns per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Implement bitmap unpacking [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot merged PR #450:
URL: https://github.com/apache/arrow-nanoarrow/pull/450
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Implement bitmap unpacking [arrow-nanoarrow]
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on code in PR #450:
URL: https://github.com/apache/arrow-nanoarrow/pull/450#discussion_r1586775118
##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -1815,6 +1815,51 @@ cdef class CBufferView:
else:
return self._iter_dispatch(offset, length)
+ def unpack_bits_into(self, dest, offset=0, length=None, dest_offset=0):
+ if self._data_type != NANOARROW_TYPE_BOOL:
+ raise ValueError("Can't unpack non-boolean buffer")
+
+ if length is None:
+ length = self.n_elements
+
+ if offset < 0 or length < 0 or (offset + length) > self.n_elements:
+ raise IndexError(
+ f"offset {offset} and length {length} do not describe a valid slice "
+ f"of buffer with {self.n_elements} elements"
+ )
+
+ cdef Py_buffer buffer
+ PyObject_GetBuffer(dest, &buffer, PyBUF_WRITABLE | PyBUF_ANY_CONTIGUOUS)
+ if buffer.itemsize != 1:
+ PyBuffer_Release(&buffer)
+ raise ValueError("Destination buffer has itemsize != 1")
+
+ if buffer.len < (dest_offset + length):
Review Comment:
Maybe worth also checking that `dest_offset` is non-negative? I think you'd get UB if a user decides to do something like `unpack_bits_into(..., len(dest) + 1, -1)` (not that they should...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Implement bitmap unpacking [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #450:
URL: https://github.com/apache/arrow-nanoarrow/pull/450#discussion_r1587972214
##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -1815,6 +1815,51 @@ cdef class CBufferView:
else:
return self._iter_dispatch(offset, length)
+ def unpack_bits_into(self, dest, offset=0, length=None, dest_offset=0):
+ if self._data_type != NANOARROW_TYPE_BOOL:
+ raise ValueError("Can't unpack non-boolean buffer")
+
+ if length is None:
+ length = self.n_elements
+
+ if offset < 0 or length < 0 or (offset + length) > self.n_elements:
+ raise IndexError(
+ f"offset {offset} and length {length} do not describe a valid slice "
+ f"of buffer with {self.n_elements} elements"
+ )
+
+ cdef Py_buffer buffer
+ PyObject_GetBuffer(dest, &buffer, PyBUF_WRITABLE | PyBUF_ANY_CONTIGUOUS)
+ if buffer.itemsize != 1:
+ PyBuffer_Release(&buffer)
+ raise ValueError("Destination buffer has itemsize != 1")
+
+ if buffer.len < (dest_offset + length):
Review Comment:
Good catch! Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org