Re: [I] [C#] IPC stream writer should write slices of buffers when writing sliced arrays [arrow]

["adamreeve (via GitHub)" <gi...@apache.org>]

adamreeve commented on issue #40517:
URL: https://github.com/apache/arrow/issues/40517#issuecomment-1996171032

   I initially thought that the problem was just that the files were bigger than they need to be, and that the array offsets would be correctly round-tripped. But on further investigation, it looks like the IPC files written from sliced arrays are invalid and can't be read back by the C# library or Python library.
   
   Reading these sliced batches back from C# fails with:
   ```
   System.IO.InvalidDataException
   Null count length must be >= 0
      at Apache.Arrow.Ipc.ArrowReaderImplementation.LoadField(MetadataVersion version, RecordBatchEnumerator& recordBatchEnumerator, Field field, FieldNode& fieldNode, ByteBuffer bodyData, IBufferCreator bufferCreator)
   ```
   
   and Python displays the arrays as invalid:
   ```
   <Invalid array: Buffer #0 too small in array of type int32 and length 100: expected at least 13 byte(s), got 0>
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org