You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Jim Pivarski (Jira)" <ji...@apache.org> on 2020/07/24 22:25:00 UTC
[jira] [Created] (ARROW-9556) Segfaults in UnionArray with null
values
Jim Pivarski created ARROW-9556:
-----------------------------------
Summary: Segfaults in UnionArray with null values
Key: ARROW-9556
URL: https://issues.apache.org/jira/browse/ARROW-9556
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 1.0.0
Environment: Conda, but pyarrow was installed using pip (in the conda environment)
Reporter: Jim Pivarski
Extracting null values from a UnionArray containing nulls and constructing a UnionArray with a bitmask in pyarrow.Array.from_buffers causes segfaults in pyarrow 1.0.0. I have an environment with pyarrow 0.17.0 and all of the following run correctly without segfaults in the older version.
Here's a UnionArray that works (because there are no nulls):
{code:java}
# GOOD
a = pyarrow.UnionArray.from_sparse(
pyarrow.array([0, 1, 0, 0, 1], type=pyarrow.int8()),
[
pyarrow.array([0.0, 1.1, 2.2, 3.3, 4.4]),
pyarrow.array([True, True, False, True, False]),
],
)
a.to_pylist(){code}
Here's one the fails when you try a.to_pylist() or even just a[2], because one of the children has a null at 2:
{code:java}
# SEGFAULT
a = pyarrow.UnionArray.from_sparse(
pyarrow.array([0, 1, 0, 0, 1], type=pyarrow.int8()),
[
pyarrow.array([0.0, 1.1, None, 3.3, 4.4]),
pyarrow.array([True, True, False, True, False]),
],
)
a.to_pylist() # also just a[2] causes a segfault{code}
Here's another that fails because both children have nulls; the segfault occurs at both positions with nulls:
{code:java}
# SEGFAULT
a = pyarrow.UnionArray.from_sparse(
pyarrow.array([0, 1, 0, 0, 1], type=pyarrow.int8()),
[
pyarrow.array([0.0, 1.1, None, 3.3, 4.4]),
pyarrow.array([True, None, False, True, False]),
],
)
a.to_pylist() # also a[1] and a[2] cause segfaults{code}
Here's one that succeeds, but it's dense, rather than sparse:
{code:java}
# GOOD
a = pyarrow.UnionArray.from_dense(
pyarrow.array([0, 1, 0, 0, 0, 1, 1], type=pyarrow.int8()),
pyarrow.array([0, 0, 1, 2, 3, 1, 2], type=pyarrow.int32()),
[pyarrow.array([0.0, 1.1, 2.2, 3.3]), pyarrow.array([True, True, False])],
)
a.to_pylist(){code}
Here's a dense that fails because one child has a null:
{code:java}
# SEGFAULT
a = pyarrow.UnionArray.from_dense(
pyarrow.array([0, 1, 0, 0, 0, 1, 1], type=pyarrow.int8()),
pyarrow.array([0, 0, 1, 2, 3, 1, 2], type=pyarrow.int32()),
[pyarrow.array([0.0, 1.1, None, 3.3]), pyarrow.array([True, True, False])],
)
a.to_pylist() # also just a[3] causes a segfault{code}
Here's a dense that fails in two positions because both children have a null:
{code:java}
# SEGFAULT
a = pyarrow.UnionArray.from_dense(
pyarrow.array([0, 1, 0, 0, 0, 1, 1], type=pyarrow.int8()),
pyarrow.array([0, 0, 1, 2, 3, 1, 2], type=pyarrow.int32()),
[pyarrow.array([0.0, 1.1, None, 3.3]), pyarrow.array([True, None, False])],
)
a.to_pylist() # also a[3] and a[5] cause segfaults{code}
In all of the above, we created the UnionArray using its from_dense method. We could instead create it with pyarrow.Array.from_buffers. If created with content0 and content1 that have no nulls, it's fine, but if created with nulls in the content, it segfaults as soon as you view the null value.
{code:java}
# GOOD
content0 = pyarrow.array([0.0, 1.1, 2.2, 3.3, 4.4])
content1 = pyarrow.array([True, True, False, True, False])
# SEGFAULT
content0 = pyarrow.array([0.0, 1.1, 2.2, None, 4.4])
content1 = pyarrow.array([True, True, False, True, False])
types = pyarrow.union(
[pyarrow.field("0", content0.type), pyarrow.field("1", content1.type)],
"sparse",
[0, 1],
)
a = pyarrow.Array.from_buffers(
types,
5,
[
None,
pyarrow.py_buffer(numpy.array([0, 1, 0, 0, 1], numpy.int8)),
],
children=[content0, content1],
)
a.to_pylist() # also just a[3] causes a segfault{code}
Similarly for a dense union.
{code:java}
# GOOD
content0 = pyarrow.array([0.0, 1.1, 2.2, 3.3])
content1 = pyarrow.array([True, True, False])
# SEGFAULT
content0 = pyarrow.array([0.0, 1.1, None, 3.3])
content1 = pyarrow.array([True, True, False])
types = pyarrow.union(
[pyarrow.field("0", content0.type), pyarrow.field("1", content1.type)],
"dense",
[0, 1],
)
a = pyarrow.Array.from_buffers(
types,
7,
[
None,
pyarrow.py_buffer(numpy.array([0, 1, 0, 0, 0, 1, 1], numpy.int8)),
pyarrow.py_buffer(numpy.array([0, 0, 1, 2, 3, 1, 2], numpy.int32)),
],
children=[content0, content1],
)
a.to_pylist() # also just a[3] causes a segfault{code}
The next segfaults are different: instead of putting the null values in the content, we put the null value in the UnionArray itself. This time, it segfaults when it is being created. It also prints some output (all of the above were silent segfaults).
{code:java}
# SEGFAULT (even to create)
content0 = pyarrow.array([0.0, 1.1, 2.2, 3.3, 4.4])
content1 = pyarrow.array([True, True, False, True, False])
types = pyarrow.union(
[pyarrow.field("0", content0.type), pyarrow.field("1", content1.type)],
"sparse",
[0, 1],
)
a = pyarrow.Array.from_buffers(
types,
5,
[
pyarrow.py_buffer(numpy.array([251], numpy.uint8)), # (11111011)
pyarrow.py_buffer(numpy.array([0, 1, 0, 0, 1], numpy.int8)),
# exepct null here -----^
# None <--- placeholder required in pyarrow 0.17.0, not 1.0.0
],
children=[content0, content1],
)
# /arrow/cpp/src/arrow/array/array_nested.cc:617: Check failed: (data_->buffers[0]) == (nullptr)
# /home/pivarski/miniconda3/envs/test-arrow/lib/python3.8/site-packages/pyarrow/libarrow.so.100(+0x4e9938)[0x7feea9937938]
# /home/pivarski/miniconda3/envs/test-arrow/lib/python3.8/site-packages/pyarrow/libarrow.so.100(_ZN5arrow4util8ArrowLogD1Ev+0xdd)[0x7feea993814d]
# /home/pivarski/miniconda3/envs/test-arrow/lib/python3.8/site-packages/pyarrow/libarrow.so.100(_ZN5arrow16SparseUnionArray7SetDataESt10shared_ptrINS_9ArrayDataEE+0x144)[0x7feea9a869a4]
# /home/pivarski/miniconda3/envs/test-arrow/lib/python3.8/site-packages/pyarrow/libarrow.so.100(_ZN5arrow16SparseUnionArrayC1ESt10shared_ptrINS_9ArrayDataEE+0x5a)[0x7feea9a86a2a]
# /home/pivarski/miniconda3/envs/test-arrow/lib/python3.8/site-packages/pyarrow/libarrow.so.100(_ZN5arrow15VisitTypeInlineINS_8internal16ArrayDataWrapperEEENS_6StatusERKNS_8DataTypeEPT_+0x9fc)[0x7feea9a5145c]
# /home/pivarski/miniconda3/envs/test-arrow/lib/python3.8/site-packages/pyarrow/libarrow.so.100(_ZN5arrow9MakeArrayERKSt10shared_ptrINS_9ArrayDataEE+0x3f)[0x7feea9a2698f]
# /home/pivarski/miniconda3/envs/test-arrow/lib/python3.8/site-packages/pyarrow/lib.cpython-38-x86_64-linux-gnu.so(+0x1c7853)[0x7feeaa998853]
# python(+0x13af9e)[0x56146ee77f9e]
# python(_PyObject_MakeTpCall+0x3bf)[0x56146ee6d30f]
# python(_PyEval_EvalFrameDefault+0x5452)[0x56146ef20602]
# python(_PyEval_EvalCodeWithName+0x260)[0x56146ef06190]
# python(PyEval_EvalCode+0x23)[0x56146ef07a03]
# python(+0x23e2f2)[0x56146ef7b2f2]
# python(+0x251082)[0x56146ef8e082]
# python(+0x1063b9)[0x56146ee433b9]
# python(PyRun_InteractiveLoopFlags+0xea)[0x56146ee43559]
# python(+0x1065f3)[0x56146ee435f3]
# python(+0x106817)[0x56146ee43817]
# python(Py_BytesMain+0x39)[0x56146ef91a19]
# /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7feeac198b97]
# python(+0x1f8807)[0x56146ef35807]
# Aborted (core dumped)
{code}
And similarly for dense.
{code:java}
# SEGFAULT (even to create)
content0 = pyarrow.array([0.0, 1.1, 2.2, 3.3])
content1 = pyarrow.array([True, True, False])
types = pyarrow.union(
[pyarrow.field("0", content0.type), pyarrow.field("1", content1.type)],
"dense",
[0, 1],
)
a = pyarrow.Array.from_buffers(
types,
7,
[
pyarrow.py_buffer(numpy.array([251], numpy.uint8)), # (11111011)
pyarrow.py_buffer(numpy.array([0, 1, 0, 0, 0, 1, 1], numpy.int8)),
pyarrow.py_buffer(numpy.array([0, 0, 1, 2, 3, 1, 2], numpy.int32)),
# exepct null here -----^
],
children=[content0, content1],
)
# /arrow/cpp/src/arrow/array/array_nested.cc:627: Check failed: (data_->buffers[0]) == (nullptr)
# /home/pivarski/miniconda3/envs/test-arrow/lib/python3.8/site-packages/pyarrow/libarrow.so.100(+0x4e9938)[0x7f2fb6ad7938]
# /home/pivarski/miniconda3/envs/test-arrow/lib/python3.8/site-packages/pyarrow/libarrow.so.100(_ZN5arrow4util8ArrowLogD1Ev+0xdd)[0x7f2fb6ad814d]
# /home/pivarski/miniconda3/envs/test-arrow/lib/python3.8/site-packages/pyarrow/libarrow.so.100(_ZN5arrow15DenseUnionArray7SetDataERKSt10shared_ptrINS_9ArrayDataEE+0x174)[0x7f2fb6c274a4]
# /home/pivarski/miniconda3/envs/test-arrow/lib/python3.8/site-packages/pyarrow/libarrow.so.100(_ZN5arrow15DenseUnionArrayC2ERKSt10shared_ptrINS_9ArrayDataEE+0x44)[0x7f2fb6c27524]
# /home/pivarski/miniconda3/envs/test-arrow/lib/python3.8/site-packages/pyarrow/libarrow.so.100(_ZN5arrow15VisitTypeInlineINS_8internal16ArrayDataWrapperEEENS_6StatusERKNS_8DataTypeEPT_+0xb14)[0x7f2fb6bf1574]
# /home/pivarski/miniconda3/envs/test-arrow/lib/python3.8/site-packages/pyarrow/libarrow.so.100(_ZN5arrow9MakeArrayERKSt10shared_ptrINS_9ArrayDataEE+0x3f)[0x7f2fb6bc698f]
# /home/pivarski/miniconda3/envs/test-arrow/lib/python3.8/site-packages/pyarrow/lib.cpython-38-x86_64-linux-gnu.so(+0x1c7853)[0x7f2fb7b38853]
# python(+0x13af9e)[0x558cf09edf9e]
# python(_PyObject_MakeTpCall+0x3bf)[0x558cf09e330f]
# python(_PyEval_EvalFrameDefault+0x5452)[0x558cf0a96602]
# python(_PyEval_EvalCodeWithName+0x260)[0x558cf0a7c190]
# python(PyEval_EvalCode+0x23)[0x558cf0a7da03]
# python(+0x23e2f2)[0x558cf0af12f2]
# python(+0x251082)[0x558cf0b04082]
# python(+0x1063b9)[0x558cf09b93b9]
# python(PyRun_InteractiveLoopFlags+0xea)[0x558cf09b9559]
# python(+0x1065f3)[0x558cf09b95f3]
# python(+0x106817)[0x558cf09b9817]
# python(Py_BytesMain+0x39)[0x558cf0b07a19]
# /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f2fb9338b97]
# python(+0x1f8807)[0x558cf0aab807]
# Aborted (core dumped){code}
It might be two distinct bugs, but they're both related to UnionArrays and nulls, and they're both newer than 0.17.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)