Streaming

[Mayan Moudgill <ma...@bestweb.net>]

I've implemented a version of client-side streaming for lists in 
cthrift. It is _fully_ compatible with the Thrift TBinaryProtocol. It 
should be possible to adapt these ideas to Thrift.

It works as follows:
1. Split the RPC into a send and a recieve part, so instead of
   foo()
  call
    foo_send()
    foo_recv()

2. For sending a stream, split into the part before the streamed list 
(including the count), a part that can send portions of the streamed 
list, and the part after the list. Thus
   foo_send_stream_start(..., len_of_list)
   while( !done ) {
     foo_send_stream_do_stream(..., len_of_chunk, chunk);
   }
   foo_send_stream_end(...)
This idea can be extended to handle multiple streams

3. For receiving a stream, the _recv() is split into two functions: the 
first which returns the count, and the other whcih returns sections of 
the stream.
   total = foo_recv_stream_len()
   n = 0;
   while( n != total ) {
     size = foo_recv_stream_data(..., max, res);
     n += size;
   }
Note that max is the maximum chunk size, while size is the size actually 
read. This can be less than max so that it can handle the case that only 
part of the stream has been transferred.

Re: Streaming

[Mayan Moudgill <ma...@bestweb.net>]

Ok, there's a little more documentation about it at:
https://sourceforge.net/apps/mediawiki/cthrift/index.php?title=Asynchrony_%26_Streaming

Still working on figuring out what the right model is for the server.

Mayan Moudgill wrote:
> 
> I've implemented a version of client-side streaming for lists in 
> cthrift. It is _fully_ compatible with the Thrift TBinaryProtocol. It 
> should be possible to adapt these ideas to Thrift.
> 
> It works as follows:
> 1. Split the RPC into a send and a recieve part, so instead of
>   foo()
>  call
>    foo_send()
>    foo_recv()
> 
> 2. For sending a stream, split into the part before the streamed list 
> (including the count), a part that can send portions of the streamed 
> list, and the part after the list. Thus
>   foo_send_stream_start(..., len_of_list)
>   while( !done ) {
>     foo_send_stream_do_stream(..., len_of_chunk, chunk);
>   }
>   foo_send_stream_end(...)
> This idea can be extended to handle multiple streams
> 
> 3. For receiving a stream, the _recv() is split into two functions: the 
> first which returns the count, and the other whcih returns sections of 
> the stream.
>   total = foo_recv_stream_len()
>   n = 0;
>   while( n != total ) {
>     size = foo_recv_stream_data(..., max, res);
>     n += size;
>   }
> Note that max is the maximum chunk size, while size is the size actually 
> read. This can be less than max so that it can handle the case that only 
> part of the stream has been transferred.
> 
> 
> 

Re: Streaming

[Mayan Moudgill <ma...@bestweb.net>]

Ok, there's a little more documentation about it at:
https://sourceforge.net/apps/mediawiki/cthrift/index.php?title=Asynchrony_%26_Streaming

Still working on figuring out what the right model is for the server.

Mayan Moudgill wrote:
> 
> I've implemented a version of client-side streaming for lists in 
> cthrift. It is _fully_ compatible with the Thrift TBinaryProtocol. It 
> should be possible to adapt these ideas to Thrift.
> 
> It works as follows:
> 1. Split the RPC into a send and a recieve part, so instead of
>   foo()
>  call
>    foo_send()
>    foo_recv()
> 
> 2. For sending a stream, split into the part before the streamed list 
> (including the count), a part that can send portions of the streamed 
> list, and the part after the list. Thus
>   foo_send_stream_start(..., len_of_list)
>   while( !done ) {
>     foo_send_stream_do_stream(..., len_of_chunk, chunk);
>   }
>   foo_send_stream_end(...)
> This idea can be extended to handle multiple streams
> 
> 3. For receiving a stream, the _recv() is split into two functions: the 
> first which returns the count, and the other whcih returns sections of 
> the stream.
>   total = foo_recv_stream_len()
>   n = 0;
>   while( n != total ) {
>     size = foo_recv_stream_data(..., max, res);
>     n += size;
>   }
> Note that max is the maximum chunk size, while size is the size actually 
> read. This can be less than max so that it can handle the case that only 
> part of the stream has been transferred.
> 
> 
>