You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "drain99 (via GitHub)" <gi...@apache.org> on 2026/04/14 10:54:50 UTC

[I] `_export_to_c` segmentation fault for `binary_view` array [arrow]

drain99 opened a new issue, #49740:
URL: https://github.com/apache/arrow/issues/49740

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   `_export_to_c` seems to cause segmentation fault for `binary_view` array constructed as follows:
   `ChunkedArray(binary...) --combine--> BinaryArray --cast--> BinaryViewArray`
   
   Reproducer below
   
   ```py
   import ctypes
   import pyarrow as pa
   
   # C Data Interface layouts
   class ArrowSchema(ctypes.Structure):
       _fields_ = [
           ("format", ctypes.c_char_p),
           ("name", ctypes.c_char_p),
           ("metadata", ctypes.c_char_p),
           ("flags", ctypes.c_int64),
           ("n_children", ctypes.c_int64),
           ("children", ctypes.POINTER(ctypes.c_void_p)),
           ("dictionary", ctypes.c_void_p),
           ("release", ctypes.c_void_p),
           ("private_data", ctypes.c_void_p),
       ]
   
   class ArrowArray(ctypes.Structure):
       _fields_ = [
           ("length", ctypes.c_int64),
           ("null_count", ctypes.c_int64),
           ("offset", ctypes.c_int64),
           ("n_buffers", ctypes.c_int64),
           ("n_children", ctypes.c_int64),
           ("buffers", ctypes.POINTER(ctypes.c_void_p)),
           ("children", ctypes.POINTER(ctypes.c_void_p)),
           ("dictionary", ctypes.c_void_p),
           ("release", ctypes.c_void_p),
           ("private_data", ctypes.c_void_p),
       ]
   
   c_array = ArrowArray()
   c_schema = ArrowSchema()
   
   arrs = [
       # exporting this is fine
       pa.array([b'a', b'e', b'a', b'e'], type=pa.binary_view()),
       # this also works fine
       pa.chunked_array([
           pa.array([b'a', b'e'], type=pa.binary_view()),
           pa.array([b'a', b'e'], type=pa.binary_view())
       ]).combine_chunks(),
       # chunked(binary...) -> binary -> binary_view segfaults
       pa.chunked_array([
           pa.array([b'a', b'e'], type=pa.binary()),
           pa.array([b'a', b'e'], type=pa.binary())
       ]).combine_chunks().cast(pa.binary_view()),
   ]
   print(arrs)
   print([arr.validate(full=True) for arr in arrs])
   arrs[0]._export_to_c(ctypes.addressof(c_array),ctypes.addressof(c_schema),)
   arrs[1]._export_to_c(ctypes.addressof(c_array),ctypes.addressof(c_schema),)
   # this fails
   arrs[2]._export_to_c(ctypes.addressof(c_array),ctypes.addressof(c_schema),)
   ```
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] `_export_to_c` segmentation fault for `binary_view` array [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #49740:
URL: https://github.com/apache/arrow/issues/49740#issuecomment-4706068908

   I agree a clarification may be warranted in https://arrow.apache.org/docs/format/CDataInterface.html#c.ArrowArray.buffers
   
   For now, it is claiming that "any buffer" can be NULL "if the size in bytes of the corresponding buffer would be 0", but that sounds a bit too lax for variadic buffers. Can you perhaps open a separate issue for that @andishgar ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] `_export_to_c` segmentation fault for `binary_view` array [arrow]

Posted by "fenfeng9 (via GitHub)" <gi...@apache.org>.
fenfeng9 commented on issue #49740:
URL: https://github.com/apache/arrow/issues/49740#issuecomment-4682809502

   @pitrou Sure, I'd be happy to take this one.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] `_export_to_c` segmentation fault for `binary_view` array [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #49740:
URL: https://github.com/apache/arrow/issues/49740#issuecomment-4304588261

   @AlenkaF Thanks for the analysis. I agree it's probably not ok to have null variadic buffers, we should instead resize the `buffers` in the cast kernel's output.
   
   We should also update our array validation routines to ensure that null variadic buffers don't pass validation (i.e., in the snippet above, `short.validate()` should fail but it currently succeeds).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] `_export_to_c` segmentation fault for `binary_view` array [arrow]

Posted by "andishgar (via GitHub)" <gi...@apache.org>.
andishgar commented on issue #49740:
URL: https://github.com/apache/arrow/issues/49740#issuecomment-4699766095

   @fenfeng9
   
   A few notes:
   
   1. `ArrowArray` can be produced in many ways, and using builders is just one of them.
   2. The most notable feature of `BinaryView`/`StringView` is that they can contain multiple data buffers. These buffers may or may not be referenced by the views, and the presence of non-deduced data buffers is entirely optional.
   3.
   
   > I also don't see the spec explicitly clarifying this, but I tend to think this should be considered an invalid state.
   
   Regarding the above comment, the Arrow specification is intended to serve as a reference for other implementations. If a null data buffer is to be considered an invalid state, I think it would be important to verify whether other reference implementations (Julia, Rust, Java, Go, Erlang, etc.) also treat it as invalid.
   
   Even if all reference implementations consider a null data buffer invalid, this behavior should be explicitly documented in the specification to avoid ambiguity for other implementations.
   
   Perhaps after @pitrou confirms whether a null data buffer can cause problems, I should open a separate issue to clarify this in the specification.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] `_export_to_c` segmentation fault for `binary_view` array [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #49740:
URL: https://github.com/apache/arrow/issues/49740#issuecomment-4851926780

   Issue resolved by pull request 50166
   https://github.com/apache/arrow/pull/50166


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] `_export_to_c` segmentation fault for `binary_view` array [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #49740: [C++][Python] `_export_to_c` segmentation fault for `binary_view` array
URL: https://github.com/apache/arrow/issues/49740


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] `_export_to_c` segmentation fault for `binary_view` array [arrow]

Posted by "andishgar (via GitHub)" <gi...@apache.org>.
andishgar commented on issue #49740:
URL: https://github.com/apache/arrow/issues/49740#issuecomment-4706333264

   @pitrou, I created the following issue for it [here](https://github.com/apache/arrow/issues/50172).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] `_export_to_c` segmentation fault for `binary_view` array [arrow]

Posted by "andishgar (via GitHub)" <gi...@apache.org>.
andishgar commented on issue #49740:
URL: https://github.com/apache/arrow/issues/49740#issuecomment-4699043298

   @fenfeng9, I'm aware that you submitted a PR. I don't intend to take it over – I'm only asking to clarify the memory layout of BinaryView.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] `_export_to_c` segmentation fault for `binary_view` array [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #49740:
URL: https://github.com/apache/arrow/issues/49740#issuecomment-4682638651

   @andishgar @fenfeng9 @Periecle Is one of you interested in working on this? While the reproducer is in Python, the bug is most probably in the C++ codebase.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] `_export_to_c` segmentation fault for `binary_view` array [arrow]

Posted by "andishgar (via GitHub)" <gi...@apache.org>.
andishgar commented on issue #49740:
URL: https://github.com/apache/arrow/issues/49740#issuecomment-4699028395

   @pitrou
   
   This is the minimum C++ code to produce the error:
   
   ```c++
   TEST(CdataInterface, BinaryView) {
     arrow::BinaryViewBuilder builder;
     ASSERT_OK(builder.Append("1"));
     ASSERT_OK_AND_ASSIGN(auto array_0, builder.Finish());
     array_0->data()->buffers.push_back(nullptr);
     struct ArrowArray array_out;
     struct ArrowSchema schema_out;
     ASSERT_OK(arrow::ExportArray(*array_0, &array_out, &schema_out));
   }
   ```
   
   The root of the problem is the line below, which assumes all data buffers of BinaryView are non‑null:
   https://github.com/apache/arrow/blob/e8b7b4e35e231a0fcdbfa74f6a6b0075108dd5dc/cpp/src/arrow/c/bridge.cc#L611
   
   I have a question: as far as I have read the specification, it does not mention whether the data buffer of BinaryViewType can be null or not. Is there any clarification for that in the Arrow specification? (If the data buffer can be null, the solution for this issue is only one line of code.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] `_export_to_c` segmentation fault for `binary_view` array [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #49740:
URL: https://github.com/apache/arrow/issues/49740#issuecomment-4295607756

   Thank you for opening the issue!
   I can reproduce the segfault locally, the repr provided is very clear. I did a bit of looking into the issue and asked Copilot for some help. What I found was that the segfault only happens for short strings and works for long strings, for example:
   
   ```python
   In [2]: long = pa.chunked_array([
      ...:     pa.array([b'a' * 13, b'e' * 13], type=pa.binary()),
      ...:     pa.array([b'a' * 13, b'e' * 13], type=pa.binary()),
      ...: ]).combine_chunks().cast(pa.binary_view())
   
   In [3]: long._export_to_c(ctypes.addressof(c_array), ctypes.addressof(c_schema))
      ...: print("Long strings: OK")
      ...: 
   Long strings: OK
   ```
   
   ```python
   In [4]: short = pa.chunked_array([
      ...:     pa.array([b'a', b'e'], type=pa.binary()),
      ...:     pa.array([b'a', b'e'], type=pa.binary()),
      ...: ]).combine_chunks().cast(pa.binary_view())
   
   In [5]: short._export_to_c(ctypes.addressof(c_array), ctypes.addressof(c_schema))
      ...: print("Short strings: OK")  # never reached
   [1]    18460 segmentation fault  ipython
   ```
   
   Copilot is pointing out that the bug is in the `scalar_cast_string.cc` implementation where the extra data buffer is dropped because `all_entries_are_inline` is `True`
   
   https://github.com/apache/arrow/blob/e8b7b4e35e231a0fcdbfa74f6a6b0075108dd5dc/cpp/src/arrow/compute/kernels/scalar_cast_string.cc#L465-L467
   
   So this would be a bug in the cast kernel. We might also update the bridge file 
   https://github.com/apache/arrow/blob/e8b7b4e35e231a0fcdbfa74f6a6b0075108dd5dc/cpp/src/arrow/c/bridge.cc#L606-L614
   guarding against null-pointer variadic buffers.
   
   cc @pitrou 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] `_export_to_c` segmentation fault for `binary_view` array [arrow]

Posted by "fenfeng9 (via GitHub)" <gi...@apache.org>.
fenfeng9 commented on issue #49740:
URL: https://github.com/apache/arrow/issues/49740#issuecomment-4689460246

   A smaller reproduction case:
   Casting short binary values to `binary_view` produces an array whose third buffer slot is `None` (`[None, 32, None]`). 
   
   And `validate(full=True)` still succeeds, but `_export_to_c` segfaults when exporting that null variadic buffer slot.
   
   ### Reproduce
   ```python
   import ctypes
   import platform
   import pyarrow as pa
   
   
   if __name__ == "__main__":
       print(f"Python {platform.python_version()}, PyArrow {pa.__version__}", flush=True)
       print(flush=True)
   
       arr = pa.array([b"a", b"e"], type=pa.binary())
       print("binary array:", flush=True)
       print(arr, flush=True)
       print(
           f"binary buffers      : {[None if buf is None else buf.size for buf in arr.buffers()]}",
           flush=True,
       )
   
       arr = arr.cast(pa.binary_view())
       print()
       print("after cast to binary_view:", flush=True)
       print(arr, flush=True)
       print(
           f"binary_view buffers : {[None if buf is None else buf.size for buf in arr.buffers()]}",
           flush=True,
       )
   
       arr.validate(full=True)
       print("validate(full=True) : OK", flush=True)
   
       print("_export_to_c        : calling", flush=True)
       out = ctypes.create_string_buffer(1024)
       arr._export_to_c(ctypes.addressof(out))
       print("_export_to_c        : OK", flush=True)
   ```
   
   ### Result
   ```python
   Python 3.12.13, PyArrow 24.0.0
   
   binary array:
   [
     61,
     65
   ]
   binary buffers      : [None, 12, 2]
   
   after cast to binary_view:
   [
     61,
     65
   ]
   binary_view buffers : [None, 32, None]
   validate(full=True) : OK
   _export_to_c        : calling
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] `_export_to_c` segmentation fault for `binary_view` array [arrow]

Posted by "fenfeng9 (via GitHub)" <gi...@apache.org>.
fenfeng9 commented on issue #49740:
URL: https://github.com/apache/arrow/issues/49740#issuecomment-4699363013

   @andishgar Thanks for looking into this, and no worries at all.
   
   My current understanding is that, for all-inline `BinaryView` / `Utf8View` values, no variadic buffer is actually needed.
   
   In practice, I found that Arrow can currently produce/accept both of these layouts:
   
   ```text
   [validity, views]
   [validity, views, zero-size variadic data buffer]
   ```
   
   For example, the normal C++ `BinaryViewBuilder::Append()` path produces the first form, while this PyArrow construction path produces the second form:
   
   ```python
   import pyarrow as pa
   
   arr = pa.array([b"ab", b"cd", b"ef"], type=pa.binary_view())
   print([None if b is None else b.size for b in arr.buffers()])
   ```
   
   ```text
   [None, 48, 0]
   ```
   
   The extra zero-size buffer seems to come from the PyArrow conversion path reserving data space before appending the value:
   
   https://github.com/apache/arrow/blob/16fe34250a2ef261790b9cc414fdf0831669cf9f/python/pyarrow/src/arrow/python/python_to_arrow.cc#L770-L775
   
   For short values, the data is still encoded inline, so the reserved heap block is unused and later becomes a zero-size variadic data buffer.
   
   For this case:
   
   ```text
   [validity, views, null variadic buffer slot]
   ```
   I also don't see the spec explicitly clarifying this, but I tend to think this should be considered an invalid state.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] `_export_to_c` segmentation fault for `binary_view` array [arrow]

Posted by "fenfeng9 (via GitHub)" <gi...@apache.org>.
fenfeng9 commented on issue #49740:
URL: https://github.com/apache/arrow/issues/49740#issuecomment-4700827129

   @andishgar  Thank you, that makes sense to me. I really appreciate the clarification.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org