You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Amarinder Randhawa <as...@gmail.com> on 2025/01/21 19:57:10 UTC
Inquiry on Using RecordBatchStreamWriter/RecordBatchStreamReader for Network Transmission
Hello Arrow developers
I’m exploring the use of RecordBatchStreamWriter and RecordBatchStreamReader
to serialize/deserialize Arrow data, over HTTP, as binary data payload. My
assumption is that these methods provide a machine-independent
serialization mechanism, allowing the Arrow format over HTTP to serve as a
reliable and efficient protocol for data exchange between client and server.
Few questions:
1. Can I depend on these interfaces to leverage Arrow format as binary
exchange mechanism over HTTP?
2. Are there any limitations or considerations I should be aware of when
adopting this approach for such use cases?
3. Is there a standardization in place for compatibility as clients
machines may update arrow library independently (example client using arrow
version 19 for StreamReader while the server is using version 18 for
StreamWriter).
Your insights and advice would be greatly appreciated.
Thanks
Amarinder S Randhawa
Re: Inquiry on Using RecordBatchStreamWriter/RecordBatchStreamReader for Network Transmission
Posted by David Li <li...@apache.org>.
> David - Although it is mentioned that the Arrow Format major version has
> not changed till now (how will the upgrade work whenever this change
> happens).
A *major* version change is extremely unlikely at this point and I don't think we can say how it would work. It would almost certainly be backwards-incompatible (else it wouldn't be a major version change), but I don't think I'd lose sleep planning for it coming anytime soon.
On Sat, Feb 1, 2025, at 03:03, Amarinder Randhawa wrote:
> Thank you Aldrin and David for sharing the helpful links.
> This really unblocked me and the
> https://github.com/apache/arrow-experiments/blob/main/http was very
> directed to speed up the use of Arrow IPC.
>
> David - Although it is mentioned that the Arrow Format major version has
> not changed till now (how will the upgrade work whenever this change
> happens).
>
> Much appreciated.
> - Amarinder S Randhawa
>
> On Tue, 21 Jan 2025 at 15:58, Aldrin <oc...@pm.me> wrote:
>
>> Hello!
>>
>> Can I depend on these interfaces to leverage Arrow format as binary
>> exchange mechanism over HTTP?
>>
>> Yes. You can see [1] for a bit of discussion and some github links. But,
>> the short answer is that the stream writer and stream reader interfaces are
>> convenient interfaces to data movement over IPC streams, and is good over
>> any medium (AFAIK).
>>
>> Is there a standardization in place for compatibility...
>>
>> Yes! It is the arrow columnar format [2] which is versioned separately
>> from any particular implementation of it. For example, the versions you're
>> talking about are for the C++ implementation.
>>
>> As for your other question, that's a bit broad and maybe can be mostly
>> addressed via a new blog post [3] or some future post in that series. Plus,
>> there's a variety of information you can find by looking around. If you
>> have a more specific question feel free to follow up on the mailing list.
>>
>>
>>
>> [1]: https://lists.apache.org/thread/886cnx6ytjst3smmytz4r4ddcbv95191
>>
>> [2]: https://arrow.apache.org/docs/format/Columnar.html#format-columnar
>>
>> [3]: https://arrow.apache.org/blog/2025/01/10/arrow-result-transfer/
>>
>>
>> Sent from Proton Mail <https://proton.me/mail/home> for iOS
>>
>>
>> On Tue, Jan 21, 2025 at 11:57, Amarinder Randhawa <asr.randhawa@gmail.com
>> <On+Tue,+Jan+21,+2025+at+11:57,+Amarinder+Randhawa+%3C%3Ca+href=>> wrote:
>>
>> Hello Arrow developers
>>
>> I’m exploring the use of RecordBatchStreamWriter and
>> RecordBatchStreamReader
>> to serialize/deserialize Arrow data, over HTTP, as binary data payload. My
>> assumption is that these methods provide a machine-independent
>> serialization mechanism, allowing the Arrow format over HTTP to serve as a
>> reliable and efficient protocol for data exchange between client and
>> server.
>>
>> Few questions:
>>
>> 1. Can I depend on these interfaces to leverage Arrow format as binary
>> exchange mechanism over HTTP?
>>
>> 2. Are there any limitations or considerations I should be aware of when
>> adopting this approach for such use cases?
>>
>> 3. Is there a standardization in place for compatibility as clients
>> machines may update arrow library independently (example client using
>> arrow
>> version 19 for StreamReader while the server is using version 18 for
>> StreamWriter).
>>
>> Your insights and advice would be greatly appreciated.
>>
>>
>> Thanks
>>
>> Amarinder S Randhawa
>>
>>
Re: Inquiry on Using RecordBatchStreamWriter/RecordBatchStreamReader for Network Transmission
Posted by Amarinder Randhawa <as...@gmail.com>.
Thank you Aldrin and David for sharing the helpful links.
This really unblocked me and the
https://github.com/apache/arrow-experiments/blob/main/http was very
directed to speed up the use of Arrow IPC.
David - Although it is mentioned that the Arrow Format major version has
not changed till now (how will the upgrade work whenever this change
happens).
Much appreciated.
- Amarinder S Randhawa
On Tue, 21 Jan 2025 at 15:58, Aldrin <oc...@pm.me> wrote:
> Hello!
>
> Can I depend on these interfaces to leverage Arrow format as binary
> exchange mechanism over HTTP?
>
> Yes. You can see [1] for a bit of discussion and some github links. But,
> the short answer is that the stream writer and stream reader interfaces are
> convenient interfaces to data movement over IPC streams, and is good over
> any medium (AFAIK).
>
> Is there a standardization in place for compatibility...
>
> Yes! It is the arrow columnar format [2] which is versioned separately
> from any particular implementation of it. For example, the versions you're
> talking about are for the C++ implementation.
>
> As for your other question, that's a bit broad and maybe can be mostly
> addressed via a new blog post [3] or some future post in that series. Plus,
> there's a variety of information you can find by looking around. If you
> have a more specific question feel free to follow up on the mailing list.
>
>
>
> [1]: https://lists.apache.org/thread/886cnx6ytjst3smmytz4r4ddcbv95191
>
> [2]: https://arrow.apache.org/docs/format/Columnar.html#format-columnar
>
> [3]: https://arrow.apache.org/blog/2025/01/10/arrow-result-transfer/
>
>
> Sent from Proton Mail <https://proton.me/mail/home> for iOS
>
>
> On Tue, Jan 21, 2025 at 11:57, Amarinder Randhawa <asr.randhawa@gmail.com
> <On+Tue,+Jan+21,+2025+at+11:57,+Amarinder+Randhawa+%3C%3Ca+href=>> wrote:
>
> Hello Arrow developers
>
> I’m exploring the use of RecordBatchStreamWriter and
> RecordBatchStreamReader
> to serialize/deserialize Arrow data, over HTTP, as binary data payload. My
> assumption is that these methods provide a machine-independent
> serialization mechanism, allowing the Arrow format over HTTP to serve as a
> reliable and efficient protocol for data exchange between client and
> server.
>
> Few questions:
>
> 1. Can I depend on these interfaces to leverage Arrow format as binary
> exchange mechanism over HTTP?
>
> 2. Are there any limitations or considerations I should be aware of when
> adopting this approach for such use cases?
>
> 3. Is there a standardization in place for compatibility as clients
> machines may update arrow library independently (example client using
> arrow
> version 19 for StreamReader while the server is using version 18 for
> StreamWriter).
>
> Your insights and advice would be greatly appreciated.
>
>
> Thanks
>
> Amarinder S Randhawa
>
>
Re: Inquiry on Using RecordBatchStreamWriter/RecordBatchStreamReader for Network Transmission
Posted by Aldrin <oc...@pm.me.INVALID>.
Hello!
Can I depend on these interfaces to leverage Arrow format as binary exchange mechanism over HTTP?Yes. You can see [1] for a bit of discussion and some github links. But, the short answer is that the stream writer and stream reader interfaces are convenient interfaces to data movement over IPC streams, and is good over any medium (AFAIK).
Is there a standardization in place for compatibility...Yes! It is the arrow columnar format [2] which is versioned separately from any particular implementation of it. For example, the versions you're talking about are for the C++ implementation.
As for your other question, that's a bit broad and maybe can be mostly addressed via a new blog post [3] or some future post in that series. Plus, there's a variety of information you can find by looking around. If you have a more specific question feel free to follow up on the mailing list.
[1]: https://lists.apache.org/thread/886cnx6ytjst3smmytz4r4ddcbv95191
[2]: https://arrow.apache.org/docs/format/Columnar.html#format-columnar
[3]: https://arrow.apache.org/blog/2025/01/10/arrow-result-transfer/
Sent from Proton Mail for iOS
On Tue, Jan 21, 2025 at 11:57, Amarinder Randhawa <asr.randhawa@gmail.com> wrote: Hello Arrow developers
I’m exploring the use of RecordBatchStreamWriter and RecordBatchStreamReader
to serialize/deserialize Arrow data, over HTTP, as binary data payload. My
assumption is that these methods provide a machine-independent
serialization mechanism, allowing the Arrow format over HTTP to serve as a
reliable and efficient protocol for data exchange between client and server.
Few questions:
1. Can I depend on these interfaces to leverage Arrow format as binary
exchange mechanism over HTTP?
2. Are there any limitations or considerations I should be aware of when
adopting this approach for such use cases?
3. Is there a standardization in place for compatibility as clients
machines may update arrow library independently (example client using arrow
version 19 for StreamReader while the server is using version 18 for
StreamWriter).
Your insights and advice would be greatly appreciated.
Thanks
Amarinder S Randhawa
Re: Inquiry on Using RecordBatchStreamWriter/RecordBatchStreamReader for Network Transmission
Posted by David Li <li...@apache.org>.
See apache/arrow-experiments [1] where there are many examples in multiple languages of doing just that, and more complicated examples of things like mixing Arrow and JSON data in a stream.
(1): Yes, that's the intent of Arrow IPC!
(2): I don't think there's anything glaring off the top of my head, except for maybe the usual points to note for using Arrow in general. (I wouldn't exchange 1 row at a time, for instance, that will have a lot of overhead.)
(3): The Arrow _format_ is standardized and versioned separately from Arrow _implementations_ so in general what you describe should work unless you are using something only available in a new format version (which happens rarely). You can read more at [2].
[1]: https://github.com/apache/arrow-experiments/tree/main/http
[2]: https://arrow.apache.org/docs/format/Versioning.html
On Wed, Jan 22, 2025, at 04:57, Amarinder Randhawa wrote:
> Hello Arrow developers
>
> I’m exploring the use of RecordBatchStreamWriter and RecordBatchStreamReader
> to serialize/deserialize Arrow data, over HTTP, as binary data payload. My
> assumption is that these methods provide a machine-independent
> serialization mechanism, allowing the Arrow format over HTTP to serve as a
> reliable and efficient protocol for data exchange between client and server.
>
> Few questions:
>
> 1. Can I depend on these interfaces to leverage Arrow format as binary
> exchange mechanism over HTTP?
>
> 2. Are there any limitations or considerations I should be aware of when
> adopting this approach for such use cases?
>
> 3. Is there a standardization in place for compatibility as clients
> machines may update arrow library independently (example client using arrow
> version 19 for StreamReader while the server is using version 18 for
> StreamWriter).
>
> Your insights and advice would be greatly appreciated.
>
>
> Thanks
>
> Amarinder S Randhawa