You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Tanguy Fautre (Jira)" <ji...@apache.org> on 2021/03/26 17:30:00 UTC

[jira] [Comment Edited] (ARROW-12100) [C#] Cannot round-trip record batch with PyArrow

    [ https://issues.apache.org/jira/browse/ARROW-12100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309594#comment-17309594 ] 

Tanguy Fautre edited comment on ARROW-12100 at 3/26/21, 5:29 PM:
-----------------------------------------------------------------

[~apitrou] Thanks for the explanation, I didn't realise there was a difference between the two.

I've replaced {{ArrowFileWriter}} with {{ArrowStreamWriter}}, but I now get the following error in Python: {{OSError: Unexpected null field Field.children in flatbuffer-encoded metadata}}.


was (Author: gpsnoopy):
[~apitrou] Thanks for the explanation, I didn't realise there was a different.

I've replaced {{ArrowFileWriter}} with {{ArrowStreamWriter}}, but I now get the following error in Python: {{OSError: Unexpected null field Field.children in flatbuffer-encoded metadata}}.

> [C#] Cannot round-trip record batch with PyArrow
> ------------------------------------------------
>
>                 Key: ARROW-12100
>                 URL: https://issues.apache.org/jira/browse/ARROW-12100
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C#, Python
>    Affects Versions: 3.0.0
>            Reporter: Tanguy Fautre
>            Priority: Major
>         Attachments: ArrowSharedMemory_20210326.zip
>
>
> Has anyone ever tried to round-trip a record batch between Arrow C# and PyArrow? I can't get PyArrow to read the data correctly.
> For context, I'm trying to do Arrow data-frames inter-process communication between C# and Python using shared memory (local TCP/IP is also an alternative). Ideally, I wouldn't even have to serialise the data and could just share the Arrow in-memory representation directly, but I'm not sure this is even possible with Apache Arrow. Full source code as attachment.
> *C#*
> {code:c#}
> using (var stream = sharedMemory.CreateStream(0, 0, MemoryMappedFileAccess.ReadWrite))
> {
>     var recordBatch = /* ... */
>     using (var writer = new ArrowFileWriter(stream, recordBatch.Schema, leaveOpen: true))
>     {
>         writer.WriteRecordBatch(recordBatch);
>         writer.WriteEnd();
>     }
> }
> {code}
> *Python*
> {code:python}
> shmem = open_shared_memory(args)
> address = get_shared_memory_address(shmem)
> buf = pa.foreign_buffer(address, args.sharedMemorySize)
> stream = pa.input_stream(buf)
> reader = pa.ipc.open_stream(stream)
> {code}
> Unfortunately, it fails with the following error: {{pyarrow.lib.ArrowInvalid: Expected to read 1330795073 metadata bytes, but only read 1230}}.
> I can see that the memory content starts with {{ARROW1\x00\x00\xff\xff\xff\xff\x08\x01\x00\x00\x10\x00\x00\x00}}. It seems that using the API calls above, PyArrow reads "ARRO" as the length of the metadata.
> I assume I'm using the API incorrectly. Has anyone got a working example?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)