You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "FlorentinD (via GitHub)" <gi...@apache.org> on 2025/10/14 13:03:41 UTC
[I] [Python][C++] Segmentation fault on serialize custom_metadata of type string [arrow]
FlorentinD opened a new issue, #47816:
URL: https://github.com/apache/arrow/issues/47816
### Describe the bug, including details regarding any error messages, version, and platform.
First of all thank you for this great library!
We are hitting a `Fatal Python error: Segmentation fault` if try to read a GET_STREAM with custom-metadata.
## Setup
1. Given a (Java) Arrow Server which on GET requests also sets custom metadata on the listener using `OutboundStreamListener::putMetadata(ArrowBuf metadata)`.
The custom-metadata is a basic String.
2. Given an pyArrow Flight client:
```
get = client.do_get(ticket)
get.read_all() # raises a Segmentation Fault
```
## Analysis
Looking at the stacktrace we could link it to the C++ method `Message::MessageImpl:Open()`.
We could verify, that even with only a single empty batch the issue occured.
In our Arrow Server implementation, we send a custom-metadata of type `String`.
The C++ Impl reads as if the custom metadata is expected to be a Map though? ([code)](https://github.com/apache/arrow/blob/be6dddf228f73dafc34dafbe1343dc6a6680eb97/cpp/src/arrow/ipc/message.cc#L105).
Omitting the String typed metadata resolves the issue.
We also know, the Java Arrow client implementation can also handle arbitrary custom metadata.
## Question
1. What format is allowed for custom metadata? Can it be also a simple string or should it always be Map?
2. If it should always be a Map, could the Python/C++ impl throw a better error message?
cc @DarthMax
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Segmentation fault on serialize custom_metadata of type string [arrow]
Posted by "FlorentinD (via GitHub)" <gi...@apache.org>.
FlorentinD commented on issue #47816:
URL: https://github.com/apache/arrow/issues/47816#issuecomment-3405206241
We are a bit lost on what is allowed for the custom metadata.
In our testing we could only avoid the segmentation fault if we dont send any custom metadata at all.
In the [docs](https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata) we can find a pattern, but its not helping us how it should look like. Also the python tests we could find did not use this pattern but just wrote a number directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Segmentation fault on serialize custom_metadata of type string [arrow]
Posted by "FlorentinD (via GitHub)" <gi...@apache.org>.
FlorentinD commented on issue #47816:
URL: https://github.com/apache/arrow/issues/47816#issuecomment-3405304391
Apparently you cannot send metadata before you send a batch?
We could workaround this by sending the metadata not on its own but along with the batch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Segmentation fault on serialize custom_metadata of type string [arrow]
Posted by "FlorentinD (via GitHub)" <gi...@apache.org>.
FlorentinD commented on issue #47816:
URL: https://github.com/apache/arrow/issues/47816#issuecomment-3405263894
stacktrace
```
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=11, no_tid=no_tid@entry=0) at pthread_kill.c:44
tid = <optimized out>
ret = 0
pd = <optimized out>
old_mask = {__val = {0}}
ret = <optimized out>
#1 0x00007f49ff898a13 in __pthread_kill_internal (threadid=<optimized out>, signo=11) at pthread_kill.c:89
No locals.
#2 0x00007f49ff83e410 in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26
ret = <optimized out>
#3 <signal handler called>
No locals.
#4 0x00007f49f8f21099 in arrow::ipc::Message::MessageImpl::Open() () from /../venv/lib/python3.13/site-packages/pyarrow/libarrow.so.1801
No symbol table info available.
#5 0x00007f49f8ee335a in arrow::ipc::Message::Open(std::shared_ptr<arrow::Buffer>, std::shared_ptr<arrow::Buffer>) () from /../venv/lib/python3.13/site-packages/pyarrow/libarrow.so.1801
No symbol table info available.
#6 0x00007f49eca0c62c in arrow::flight::internal::FlightData::OpenMessage() () from /../venv/lib/python3.13/site-packages/pyarrow/libarrow_flight.so.1801
No symbol table info available.
#7 0x00007f49eca4ad86 in arrow::flight::IpcMessageReader::ReadNextMessage() () from /../venv/lib/python3.13/site-packages/pyarrow/libarrow_flight.so.1801
No symbol table info available.
#8 0x00007f49f8f34cd5 in arrow::ipc::RecordBatchStreamReaderImpl::ReadNext() () from /../venv/lib/python3.13/site-packages/pyarrow/libarrow.so.1801
No symbol table info available.
#9 0x00007f49f8f1c67c in arrow::ipc::RecordBatchStreamReaderImpl::ReadNext(std::shared_ptr<arrow::RecordBatch>*) () from /../venv/lib/python3.13/site-packages/pyarrow/libarrow.so.1801
No symbol table info available.
#10 0x00007f49eca4b223 in arrow::flight::ClientStreamReader::Next() () from /../venv/lib/python3.13/site-packages/pyarrow/libarrow_flight.so.1801
No symbol table info available.
#11 0x00007f49eca4b7bc in arrow::flight::ClientStreamReader::ToRecordBatches(arrow::StopToken const&) () from /../venv/lib/python3.13/site-packages/pyarrow/libarrow_flight.so.1801
No symbol table info available.
#12 0x00007f49eca3a565 in arrow::flight::FlightStreamReader::ToTable(arrow::StopToken const&) () from /../venv/lib/python3.13/site-packages/pyarrow/libarrow_flight.so.1801
No symbol table info available.
#13 0x00007f49ed8fb5e4 in __pyx_pw_7pyarrow_7_flight_18FlightStreamReader_3read_all(_object*, _object* const*, long, _object*) () from /../venv/lib/python3.13/site-packages/pyarrow/_flight.cpython-313-x86_64-linux-gnu.so
No symbol table info available.
#14 0x00007f49ffd645bd in _PyObject_VectorcallTstate (tstate=0x7f4a00132df0 <_PyRuntime+283024>, callable=0x7f49eda34ad0, args=0x7f4a002e58e0, nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:168
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org