[jira] [Created] (ARROW-12100) [C#] Cannot round-trip record batch with PyArrow

["Tanguy Fautre (Jira)" <ji...@apache.org>]

Tanguy Fautre created ARROW-12100:
-------------------------------------

             Summary: [C#] Cannot round-trip record batch with PyArrow
                 Key: ARROW-12100
                 URL: https://issues.apache.org/jira/browse/ARROW-12100
             Project: Apache Arrow
          Issue Type: Bug
          Components: C#, Python
    Affects Versions: 3.0.0
            Reporter: Tanguy Fautre
         Attachments: ArrowSharedMemory_20210326.zip

Has anyone ever tried to round-trip a record batch between Arrow C# and PyArrow? I can't get PyArrow to read the data correctly.

For context, I'm trying to do Arrow data-frames inter-process communication between C# and Python using shared memory (local TCP/IP is also an alternative). Ideally, I wouldn't even have to serialise the data and could just share the Arrow in-memory representation directly, but I'm not sure this is even possible with Apache Arrow.

```c#
using (var stream = sharedMemory.CreateStream(0, 0, MemoryMappedFileAccess.ReadWrite))
{
    var recordBatch = new RecordBatch.Builder()
        .Append("Column A", false, col => col.Int32(array => array.AppendRange(Enumerable.Range(0, 10))))
        .Append("Column B", false, col => col.Float(array => array.AppendRange(Enumerable.Range(0, 10).Select(x => Convert.ToSingle(x * 2)))))
        .Append("Column C", false, col => col.String(array => array.AppendRange(Enumerable.Range(0, 10).Select(x => $"Item {x + 1}"))))
        .Append("Column D", false, col => col.Boolean(array => array.AppendRange(Enumerable.Range(0, 10).Select(x => x % 2 == 0))))
        .Build();

    using (var writer = new ArrowFileWriter(stream, recordBatch.Schema, leaveOpen: true))
    {
        writer.WriteRecordBatch(recordBatch);
        writer.WriteEnd();
    }
}
```

```python
shmem = open_shared_memory(args)
address = get_shared_memory_address(shmem)
buf = pa.foreign_buffer(address, args.sharedMemorySize)
stream = pa.input_stream(buf)
reader = pa.ipc.open_stream(stream)
```

Unfortunately, it fails with the following error: `pyarrow.lib.ArrowInvalid: Expected to read 1330795073 metadata bytes, but only read 1230`.

I can see that the memory content starts with `'ARROW1\x00\x00\xff\xff\xff\xff\x08\x01\x00\x00\x10\x00\x00\x00`. It seems that using the API calls above, PyArrow reads `ARRO` as the length of the metadata.

I assume I'm using the API incorrectly. Has anyone got a working example?

I'm attaching the full source code to this ticket.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)