You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "eitsupi (via GitHub)" <gi...@apache.org> on 2025/04/16 04:27:59 UTC

[I] Serialized polars DataFrame cannot be processed properly [arrow-nanoarrow]

eitsupi opened a new issue, #743:
URL: https://github.com/apache/arrow-nanoarrow/issues/743

   I don't know if this is a bug in polars or a problem with nanoarrow, but I think it could be a bug in nanoarrow, since Arrow C++, the reference implementation of Apache Arrow, seems to read it correctly (I'm using I noticed this bug on R, so it is not a Python problem).
   
   The binary representation of the Polars DataFrame should be an uncompressed Arrow IPC stream, but nanoarrow does not seem to read this correctly.
   
   ```python
   >>> import polars as pl
   >>> import nanoarrow as na
   >>> import pyarrow as pa
   >>> 
   >>> df = pl.DataFrame({"foo": [1, 2, 3]})
   >>> bytes = df.serialize()
   >>> 
   >>> pa.ipc.open_stream(bytes).read_all() # Ok
   pyarrow.Table
   foo: int64
   ----
   foo: [[1,2,3]]
   >>> na.ArrayStream.from_readable(bytes).to_pylist() # Error
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/home/username/.local/lib/python3.12/site-packages/nanoarrow/visitor.py", line 51, in to_pylist
       return ToPyListConverter.visit(self)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/username/.local/lib/python3.12/site-packages/nanoarrow/visitor.py", line 233, in visit
       for array in stream:
     File "src/nanoarrow/_array_stream.pyx", line 249, in nanoarrow._array_stream.CArrayStream.__next__
     File "src/nanoarrow/_array_stream.pyx", line 238, in nanoarrow._array_stream.CArrayStream.get_next
     File "src/nanoarrow/_utils.pyx", line 155, in nanoarrow._utils.Error.raise_message_not_ok
     File "src/nanoarrow/_utils.pyx", line 149, in nanoarrow._utils.Error.raise_message
   nanoarrow._utils.NanoarrowException: ArrowArrayStream::get_next() failed (22): Message flatbuffer verification failed (16) vector header out of range or unaligned
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Serialized polars DataFrame cannot be processed properly [arrow-nanoarrow]

Posted by "eitsupi (via GitHub)" <gi...@apache.org>.
eitsupi commented on issue #743:
URL: https://github.com/apache/arrow-nanoarrow/issues/743#issuecomment-2812518381

   Thanks for letting me know about that!
   I will point it out to the Polars folks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Serialized polars DataFrame cannot be processed properly [arrow-nanoarrow]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #743:
URL: https://github.com/apache/arrow-nanoarrow/issues/743#issuecomment-2809586694

   I believe this is because of flatbuffers, which we had to fix to get arrow-rs to work with nanoarrow! Upgrading flatbuffers to 24.12.23 and regenerating the generated code should fix the issue (see https://github.com/apache/arrow-rs/pull/6426 !)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Serialized polars DataFrame cannot be processed properly [arrow-nanoarrow]

Posted by "eitsupi (via GitHub)" <gi...@apache.org>.
eitsupi commented on issue #743:
URL: https://github.com/apache/arrow-nanoarrow/issues/743#issuecomment-2912037764

   I looked at this and found that the Arrow format used inside Polars is not generated by the official flatbuffers implementation, but by the planus CLI.
   I re-generated it with the latest planus CLI (pola-rs/arrow-format#2) and that seems to have fixed the problem (I don't understand what happened).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Serialized polars DataFrame cannot be processed properly [arrow-nanoarrow]

Posted by "eitsupi (via GitHub)" <gi...@apache.org>.
eitsupi closed issue #743: Serialized polars DataFrame cannot be processed properly
URL: https://github.com/apache/arrow-nanoarrow/issues/743


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org