You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "kronic (via GitHub)" <gi...@apache.org> on 2026/04/28 07:01:30 UTC
[I] Add support System.Buffers.IBufferWriter [arrow-dotnet]
kronic opened a new issue, #341:
URL: https://github.com/apache/arrow-dotnet/issues/341
### Describe the enhancement requested
Please add support Add support System.Buffers.IBufferWriter<T>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support System.Buffers.IBufferWriter [arrow-dotnet]
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on issue #341:
URL: https://github.com/apache/arrow-dotnet/issues/341#issuecomment-4334508684
Hi @kronic, could you please provide some more detail on what you mean by this. Which types should implement this interface? And can you provide an example of how you would use this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support System.Buffers.IBufferWriter [arrow-dotnet]
Posted by "kronic (via GitHub)" <gi...@apache.org>.
kronic commented on issue #341:
URL: https://github.com/apache/arrow-dotnet/issues/341#issuecomment-4334764029
Hi @adamreeve. I meant the ArrowStreamWriter constructor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support System.Buffers.IBufferWriter [arrow-dotnet]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on issue #341:
URL: https://github.com/apache/arrow-dotnet/issues/341#issuecomment-4334840535
This could be interesting, but I wonder if for full effect it might need to be threaded through the FlatBuffers layer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support System.Buffers.IBufferWriter [arrow-dotnet]
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on issue #341:
URL: https://github.com/apache/arrow-dotnet/issues/341#issuecomment-4348412111
Given we vendor the Flatbuffers code, we could modify the builder to work with an `IBufferWriter`. But other projects could probably benefit from this too, and diverging from upstream Flatbuffers too much could cause problems for us later, so it would be nice to get this added to the upstream Flatbuffers project.
I think it might be possible to do this in a backwards compatible way by refactoring `FlatBufferBuilder` to work with an `IBufferWriter`, and `ByteBuffer` could implement `IBufferWriter`, or a wrapper class added to implement it.
I think the next step would be to raise this as an issue in the Flatbuffers project and see what their thoughts are. There's probably not much point in adding explicit support for `IBufferWriter` without Flatbuffer integration as handling of most other writes like Arrow buffers doesn't require an intermediate buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support System.Buffers.IBufferWriter [arrow-dotnet]
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on issue #341:
URL: https://github.com/apache/arrow-dotnet/issues/341#issuecomment-4340037101
Yeah I'm not sure how feasible it would be for Flatbuffers to work with an IBufferWriter, but maybe that's not required. The Flatbuffers library currently creates a ReadOnlyMemory buffer that we write to the stream, so we could keep doing something similar.
I see there's an [`IBufferWriterExtensions.AsStream` method](https://learn.microsoft.com/en-us/dotnet/api/microsoft.toolkit.highperformance.ibufferwriterextensions.asstream?view=win-comm-toolkit-dotnet-7.1) in the `CommunityToolkit.HighPerformance` NuGet package, so you could use that to create a stream from an `IBufferWriter` to pass to an `ArrowStreamWriter`. Is that sufficient for your needs?
I did a quick test and this works:
```c#
using var originalBatch = TestData.CreateSampleRecordBatch(length: 100);
var bufferWriter = new ArrayBufferWriter<byte>(1024);
using (var stream = bufferWriter.AsStream())
{
using var writer = new ArrowStreamWriter(stream, originalBatch.Schema);
writer.WriteRecordBatch(originalBatch);
writer.WriteEnd();
}
using var readStream = bufferWriter.WrittenMemory.AsStream();
using (var reader = new ArrowStreamReader(readStream))
{
using var newBatch = reader.ReadNextRecordBatch();
ArrowReaderVerifier.CompareBatches(originalBatch, newBatch);
}
```
If you'd rather not add a new dependency, the implementation is pretty simple so you could also create your own wrapper class that inherits from `Stream`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support System.Buffers.IBufferWriter [arrow-dotnet]
Posted by "kronic (via GitHub)" <gi...@apache.org>.
kronic commented on issue #341:
URL: https://github.com/apache/arrow-dotnet/issues/341#issuecomment-4341699000
@adamreeve I know about this. I would like to record directly without additional copies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support System.Buffers.IBufferWriter [arrow-dotnet]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on issue #341:
URL: https://github.com/apache/arrow-dotnet/issues/341#issuecomment-4340663349
The main advantage of `IBufferWriter<byte>` is that in principle it can reduce the number of copies made of data and remove intermediate allocations. I suspect that `IBufferWriterExtensions.AsStream` helps with one of these by using an underlying sequence of buffers instead of a single long buffer that needs to be reallocated when its end is reached, but it wouldn't avoid the need for an intermediate buffer to e.g. write a serialized int32. But if you really wanted to "max out" that pattern, I think you'd also want to extend it to the FlatBuffers layer -- which today uses a temporary resizable `ByteBuffer` whose semantics I think don't translate that wall to `IBufferWriter`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org