[jira] [Commented] (ARROW-4997) [C#] ArrowStreamReader doesn't consume whole stream and doesn't implement sync read

["Eric Erhardt (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799169#comment-16799169 ] 

Eric Erhardt commented on ARROW-4997:
-------------------------------------

[~wesmckinn], [~xhochy], [~kou] - this is a pretty major issue in the C# library that is blocking usage of the Arrow C# library over a network stream. I have a [PR|https://github.com/apache/arrow/pull/4017] up to fix it that has been approved by Stephen Toub. I'm hoping this PR can be merged for the 0.13 release, as it is super important for our usages.

> [C#] ArrowStreamReader doesn't consume whole stream and doesn't implement sync read
> -----------------------------------------------------------------------------------
>
>                 Key: ARROW-4997
>                 URL: https://issues.apache.org/jira/browse/ARROW-4997
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C#
>            Reporter: Eric Erhardt
>            Assignee: Eric Erhardt
>            Priority: Blocker
>              Labels: pull-request-available
>   Original Estimate: 4h
>          Time Spent: 1h 10m
>  Remaining Estimate: 2h 50m
>
> There are 2 major issues with the ArrowStreamReader that are blocking me from using it.
>  # When it reads a batch from a .NET Stream that doesn't return the whole chunk of memory in one "Read" call (like a socket/network stream), it only calls Read once, and then continues on. This is an issue because it has "garbage" at the end of its buffer (which was never written to by the stream), and when attempting to read the next batch, it is in the middle of the previous batch from the .NET Stream. This causes all sorts of issues because it assumes the next 4 bytes are the message length, which it obviously isn't. See [the reading code|https://github.com/apache/arrow/blob/13fd813445b4738cbebbd137490fe3c02071c04b/csharp/src/Apache.Arrow/Ipc/ArrowStreamReaderImplementation.cs#L90-L97] for where it only calls Read once - it should be in a loop.
>  # ArrowStreamReader has a synchronous ReadNextRecordBatch() method - but it throws NotImplementedException. This is necessary when a caller isn't in an async method, they can't/shouldn't call the async API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)