You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "fiktor (via GitHub)" <gi...@apache.org> on 2024/04/26 02:28:29 UTC
[I] Conversion from `np.ndarray` with bytes data type to binary pyarrow array discards all bytes after the first null byte. [arrow]
fiktor opened a new issue, #41388:
URL: https://github.com/apache/arrow/issues/41388
### Describe the bug, including details regarding any error messages, version, and platform.
## Example 1
```
import numpy as np
import pyarrow as pa
np_array = np.array([b'\x00Very important data. Keep it safe!', b'Less important data. Keep it safe??'])
table = pa.Table.from_pydict({'sample': np_array})
print(table['sample'][0].as_py())
```
Expected output `b'\x00Very important data. Keep it safe!'`, actual output `b''`. Note: `type(table['sample'])` in this example is `pyarrow.lib.ChunkedArray`. `table['sample'][1]` is `b'Less important data. Keep it safe??'` as expected. The byte arrays in this example look like human-readable strings (except for the first null byte), but in reality we are trying to encode binary data as a fixed length byte array and store it in (variable length for compatibility with HuggingFace) byte array parquet column.
## Example 2
With the same `np_array`:
```
pa_array = pa.array(np_array, type=pa.binary())
print(pa_array[0])
```
Expected output `b'\x00Very important data. Keep it safe!'`, actual output `b''`. Note: `type(pa_array)` is `pyarrow.lib.BinaryArray`.
## Possible workaround
Replacing `table = pa.Table.from_pydict({'sample': np_array})` with `table = pa.Table.from_pydict({'sample': a.array(array, type=pa.binary(array.itemsize)).cast(target_type=pa.binary())})` seems to produce the intended result.
Note that in our usecase we want the resulting dtype to be `pa.binary()` (BYTE_ARRAY in parquet), not `pa.binary(35) (FIXED_LEN_BYTE_ARRAY in parquet), because the latter does not seem to be supported by HuggingFace datasets library.
## System info
The above outputs are on Ubuntu 22.04.4 LTS with Python 3.10.11, pyarrow version 12.0.1 (using parquet-cpp-arrow version 12.0.1).
## Related issues
This seems to be similar but distinct from #36308 , where the input contained byte arrays of different lengths and a conversion to `pa.string()` column was desired.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Conversion from `np.ndarray` with bytes data type to binary pyarrow array discards all bytes after the first null byte. [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #41388:
URL: https://github.com/apache/arrow/issues/41388#issuecomment-2088180177
While certainly confusing, this is actually somewhat intentional behaviour AFAIK.
NumPy explicitly describes their `S` dtype as "zero-terminated bytes" ([docs](https://numpy.org/doc/stable/reference/arrays.dtypes.html#specifying-and-constructing-data-types)). Although those dtypes are fixed-width, numpy essentially kind of supports variable length bytes/strings by padding shorter strings with null bytes. In their recent documentation, this is explained better (https://numpy.org/devdocs/user/basics.types.html#data-types-for-strings-and-bytes):
> If we specify a shorter or longer data type, the string is either truncated or zero-padded to fit in the specified width
As an example, consider this simple array:
```
>>> np_arr = np.array([b"short", b"longer one"])
>>> np_arr
array([b'short', b'longer one'], dtype='|S10')
>>> arr = pa.array(np_arr)
>>> arr
<pyarrow.lib.BinaryArray object at 0x7fba6640a8c0>
[
73686F7274,
6C6F6E676572206F6E65
]
>>> arr.to_pylist()
[b'short', b'longer one']
```
If we would not discard null bytes, then this would result in actual null bytes in the data:
```
>>> arr = pa.array(np_arr, pa.binary(10))
>>> arr.to_pylist()
[b'short\x00\x00\x00\x00\x00', b'longer one']
```
As far as I understand, that is also the reason why we still default to our variable sized string/binary types when converting from numpy's fixed-width dtypes.
And as you noted, a current workaround is indeed to specify to convert the numpy array to a fixed-size binary type.
Now, the way we drop those padded null bytes, is by using `strnlen` which I think essentially means we are searching for the first null byte to determine the length of the value:
https://github.com/apache/arrow/blob/22f88fa4a8f5ac7250f1845aace5a78d20006ef2/python/pyarrow/src/arrow/python/numpy_to_arrow.cc#L562-L566
So I assume we could change this to only drop actually _trailing_ null bytes, instead of truncating it at the first null byte we encounter (I assume numpy does this since it prints the null byte in the middle just fine?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org