You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by Jonas Ruesch <jr...@jruesch.ch> on 2016/04/29 16:16:30 UTC
Re: TZLibTransport for CSharp?
Hi Randy, Jens,
Following your suggestions and the existing Java implementation, we had a
closer look at a possible C# implementation. Below I add a few technical
notes on what we found. So far we didn't pursue any of the options further,
because for our specific application we found that in practice the
uncompressed binary format outperformed the compressed communication in
terms of time per call (tested with C++ client & server).
Technical notes on possible TZlibTransport implementations in C#/.Net
(credit: Mario G.):
Dot net supports by default a compression format which is compatible to
ZLib. It is little effort to create some wrappers to implement the
TZlibTransport for DotNet. However, the issue is that Thrift is forcing to
send data by flushing the stream, but the DotNet compressions streams do
not support flushing. This means that data is only being sent if there is
enough data available to compress the next block, or if the stream is
closed.
Thrift only closes the sending stream after the response has been received.
But as the data has never been sent, data is never received too.
To solve this issue the compression stream can be closed instead of
flushed. This works as long as it is guaranteed that thrift never sends any
additional data after calling flush. We didn't investigate if this is
guaranteed (todo).
If the above condition is not guaranteed, flush support could be
implemented by using the DotNetZip library which supports the sync_flush
mode (http://www.bolet.org/~pornin/deflate-flush.html). This mode is only
supported on a very low level, thus a lot of handling has to be implemented
manually.
(Sorry, this answer might be better suited for the developer mailing list,
but I kept it in this thread for better context.)
Best,
Jonas
On 15 February 2016 at 19:30, Randy Abernethy <ra...@apache.org> wrote:
> Hey Jonas,
>
> I think .Net has support for deflate built in, so there's not need for a
> third party lib. More info here:
>
> https://msdn.microsoft.com/en-us/library/system.io.compression.deflatestream(v=vs.110).aspx
>
> Should be pretty easy to use the Java or C++ TZlibTrans as a model and get
> the same going in C#.
>
> Best,
> Randy
>
>
> On Sun, Feb 14, 2016 at 11:59 PM, Jonas Ruesch <jr...@jruesch.ch> wrote:
>
> > Hi Jens, Randy,
> >
> > Thanks for confirming.
> > Any preference or previous experiences with a zlib-compatible zip library
> > for c#? That one looks promising: http://dotnetzip.codeplex.com/
> >
> > Cheers,
> > Jonas
> >
> >
> > On 13 February 2016 at 00:36, Jens Geyer <je...@hotmail.com> wrote:
> >
> > > Hi Jonas,
> > >
> > > patches and pull requests are welcome.
> > >
> > > Because the documentation states 'zlib transport not available for
> java'
> > >> (in the meantime it seems to be available though) I had hopes that
> there
> > >> is
> > >> a C# implementation.
> > >>
> > >
> > > That's the problem with logic. It sometimes hits you where you expect
> it
> > > the least. Just kidding ... ;-) As I said on SO, some things do not
> exist
> > > simply because nobody needed them so far. But that does not mean it
> can't
> > > or shouldn't be added if someone does need it. Especially TZlib is a
> good
> > > thing to have.
> > >
> > > Looking forward to your contribution. If you need help with it, just
> ask,
> > > and someone will answer.
> > >
> > > Have fun,
> > > JensG
> > >
> > >
> > > -----Ursprüngliche Nachricht----- From: Randy Abernethy
> > > Sent: Friday, February 12, 2016 3:48 PM
> > > To: user@thrift.apache.org
> > > Subject: Re: TZLibTransport for CSharp?
> > >
> > >
> > > Hi Jonas,
> > >
> > > The current master does not have a TZlibTransport for C#. Would be a
> > great
> > > add though! Patches always welcome.
> > >
> > > Best,
> > > Randy
> > >
> > > On Fri, Feb 12, 2016 at 2:46 AM, Jonas Ruesch <jr...@jruesch.ch>
> > wrote:
> > >
> > > Hi,
> > >>
> > >> Is there an implementation of the zlib Transport in C# (Thrift
> 0.9.3)? I
> > >> looked for it in the lib/csharp source code and believe to have
> searched
> > >> the web extensively to no avail.
> > >>
> > >> Because the documentation states 'zlib transport not available for
> java'
> > >> (in the meantime it seems to be available though) I had hopes that
> there
> > >> is
> > >> a C# implementation.
> > >>
> > >> Sorry if this question was asked before (couldn't find any traces).
> > >>
> > >> Many thanks for your hints.
> > >>
> > >>
> > >
> >
>
Re: TZLibTransport for CSharp?
Posted by Jonas Ruesch <jr...@jruesch.ch>.
Hi Jens,
To assess the performance (with respect to time) we tested only the
existing C++ TZlibTransport implementation (client and server both C++,
running on the same machine, communicating via tcp). In our current
productive setup one side is always C++, the other C# (or C++).
Cheers,
Jonas
On 29 April 2016 at 19:48, Jens Geyer <je...@hotmail.com> wrote:
> Hi Jonas,
>
> short question: What is your (non-C#) counterpart that you use to test
> against?
>
> Thanks,
> JensG
>
>
>
> -----Ursprüngliche Nachricht----- From: Randy Abernethy
> Sent: Friday, April 29, 2016 5:57 PM
>
> To: user@thrift.apache.org
> Subject: Re: TZLibTransport for CSharp?
>
> Hey Jonas,
>
> Thanks for the detailed write up! Will provide great guidance for anyone
> interested in implementing TZlibTransport for C#. My experience concurs
> with yours, using ZLib for thrift file compression can be useful but it is
> not generally an uptick for RPC. Perhaps in that context lack of flush is
> acceptable.
>
> -Randy
>
> On Fri, Apr 29, 2016 at 7:16 AM, Jonas Ruesch <jr...@jruesch.ch> wrote:
>
> Hi Randy, Jens,
>>
>> Following your suggestions and the existing Java implementation, we had a
>> closer look at a possible C# implementation. Below I add a few technical
>> notes on what we found. So far we didn't pursue any of the options
>> further,
>> because for our specific application we found that in practice the
>> uncompressed binary format outperformed the compressed communication in
>> terms of time per call (tested with C++ client & server).
>>
>> Technical notes on possible TZlibTransport implementations in C#/.Net
>> (credit: Mario G.):
>>
>> Dot net supports by default a compression format which is compatible to
>> ZLib. It is little effort to create some wrappers to implement the
>> TZlibTransport for DotNet. However, the issue is that Thrift is forcing to
>> send data by flushing the stream, but the DotNet compressions streams do
>> not support flushing. This means that data is only being sent if there is
>> enough data available to compress the next block, or if the stream is
>> closed.
>>
>> Thrift only closes the sending stream after the response has been
>> received.
>> But as the data has never been sent, data is never received too.
>>
>> To solve this issue the compression stream can be closed instead of
>> flushed. This works as long as it is guaranteed that thrift never sends
>> any
>> additional data after calling flush. We didn't investigate if this is
>> guaranteed (todo).
>>
>> If the above condition is not guaranteed, flush support could be
>> implemented by using the DotNetZip library which supports the sync_flush
>> mode (http://www.bolet.org/~pornin/deflate-flush.html). This mode is only
>> supported on a very low level, thus a lot of handling has to be
>> implemented
>> manually.
>>
>> (Sorry, this answer might be better suited for the developer mailing list,
>> but I kept it in this thread for better context.)
>>
>> Best,
>> Jonas
>>
>>
>>
>>
>> On 15 February 2016 at 19:30, Randy Abernethy <ra...@apache.org> wrote:
>>
>> > Hey Jonas,
>> >
>> > I think .Net has support for deflate built in, so there's not need for a
>> > third party lib. More info here:
>> >
>> >
>>
>> https://msdn.microsoft.com/en-us/library/system.io.compression.deflatestream(v=vs.110).aspx
>> >
>> > Should be pretty easy to use the Java or C++ TZlibTrans as a model and
>> get
>> > the same going in C#.
>> >
>> > Best,
>> > Randy
>> >
>> >
>> > On Sun, Feb 14, 2016 at 11:59 PM, Jonas Ruesch <jr...@jruesch.ch>
>> wrote:
>> >
>> > > Hi Jens, Randy,
>> > >
>> > > Thanks for confirming.
>> > > Any preference or previous experiences with a zlib-compatible zip
>> library
>> > > for c#? That one looks promising: http://dotnetzip.codeplex.com/
>> > >
>> > > Cheers,
>> > > Jonas
>> > >
>> > >
>> > > On 13 February 2016 at 00:36, Jens Geyer <je...@hotmail.com>
>> wrote:
>> > >
>> > > > Hi Jonas,
>> > > >
>> > > > patches and pull requests are welcome.
>> > > >
>> > > > Because the documentation states 'zlib transport not available for
>> > java'
>> > > >> (in the meantime it seems to be available though) I had hopes that
>> > there
>> > > >> is
>> > > >> a C# implementation.
>> > > >>
>> > > >
>> > > > That's the problem with logic. It sometimes hits you where you > >
>> > expect
>> > it
>> > > > the least. Just kidding ... ;-) As I said on SO, some things do not
>> > exist
>> > > > simply because nobody needed them so far. But that does not mean it
>> > can't
>> > > > or shouldn't be added if someone does need it. Especially TZlib is a
>> > good
>> > > > thing to have.
>> > > >
>> > > > Looking forward to your contribution. If you need help with it, just
>> > ask,
>> > > > and someone will answer.
>> > > >
>> > > > Have fun,
>> > > > JensG
>> > > >
>> > > >
>> > > > -----Ursprüngliche Nachricht----- From: Randy Abernethy
>> > > > Sent: Friday, February 12, 2016 3:48 PM
>> > > > To: user@thrift.apache.org
>> > > > Subject: Re: TZLibTransport for CSharp?
>> > > >
>> > > >
>> > > > Hi Jonas,
>> > > >
>> > > > The current master does not have a TZlibTransport for C#. Would be a
>> > > great
>> > > > add though! Patches always welcome.
>> > > >
>> > > > Best,
>> > > > Randy
>> > > >
>> > > > On Fri, Feb 12, 2016 at 2:46 AM, Jonas Ruesch <jr...@jruesch.ch>
>> > > wrote:
>> > > >
>> > > > Hi,
>> > > >>
>> > > >> Is there an implementation of the zlib Transport in C# (Thrift
>> > 0.9.3)? I
>> > > >> looked for it in the lib/csharp source code and believe to have
>> > searched
>> > > >> the web extensively to no avail.
>> > > >>
>> > > >> Because the documentation states 'zlib transport not available for
>> > java'
>> > > >> (in the meantime it seems to be available though) I had hopes that
>> > there
>> > > >> is
>> > > >> a C# implementation.
>> > > >>
>> > > >> Sorry if this question was asked before (couldn't find any traces).
>> > > >>
>> > > >> Many thanks for your hints.
>> > > >>
>> > > >>
>> > > >
>> > >
>> >
>>
>>
>
Re: TZLibTransport for CSharp?
Posted by Jens Geyer <je...@hotmail.com>.
Hi Jonas,
short question: What is your (non-C#) counterpart that you use to test
against?
Thanks,
JensG
-----Ursprüngliche Nachricht-----
From: Randy Abernethy
Sent: Friday, April 29, 2016 5:57 PM
To: user@thrift.apache.org
Subject: Re: TZLibTransport for CSharp?
Hey Jonas,
Thanks for the detailed write up! Will provide great guidance for anyone
interested in implementing TZlibTransport for C#. My experience concurs
with yours, using ZLib for thrift file compression can be useful but it is
not generally an uptick for RPC. Perhaps in that context lack of flush is
acceptable.
-Randy
On Fri, Apr 29, 2016 at 7:16 AM, Jonas Ruesch <jr...@jruesch.ch> wrote:
> Hi Randy, Jens,
>
> Following your suggestions and the existing Java implementation, we had a
> closer look at a possible C# implementation. Below I add a few technical
> notes on what we found. So far we didn't pursue any of the options
> further,
> because for our specific application we found that in practice the
> uncompressed binary format outperformed the compressed communication in
> terms of time per call (tested with C++ client & server).
>
> Technical notes on possible TZlibTransport implementations in C#/.Net
> (credit: Mario G.):
>
> Dot net supports by default a compression format which is compatible to
> ZLib. It is little effort to create some wrappers to implement the
> TZlibTransport for DotNet. However, the issue is that Thrift is forcing to
> send data by flushing the stream, but the DotNet compressions streams do
> not support flushing. This means that data is only being sent if there is
> enough data available to compress the next block, or if the stream is
> closed.
>
> Thrift only closes the sending stream after the response has been
> received.
> But as the data has never been sent, data is never received too.
>
> To solve this issue the compression stream can be closed instead of
> flushed. This works as long as it is guaranteed that thrift never sends
> any
> additional data after calling flush. We didn't investigate if this is
> guaranteed (todo).
>
> If the above condition is not guaranteed, flush support could be
> implemented by using the DotNetZip library which supports the sync_flush
> mode (http://www.bolet.org/~pornin/deflate-flush.html). This mode is only
> supported on a very low level, thus a lot of handling has to be
> implemented
> manually.
>
> (Sorry, this answer might be better suited for the developer mailing list,
> but I kept it in this thread for better context.)
>
> Best,
> Jonas
>
>
>
>
> On 15 February 2016 at 19:30, Randy Abernethy <ra...@apache.org> wrote:
>
> > Hey Jonas,
> >
> > I think .Net has support for deflate built in, so there's not need for a
> > third party lib. More info here:
> >
> >
> https://msdn.microsoft.com/en-us/library/system.io.compression.deflatestream(v=vs.110).aspx
> >
> > Should be pretty easy to use the Java or C++ TZlibTrans as a model and
> get
> > the same going in C#.
> >
> > Best,
> > Randy
> >
> >
> > On Sun, Feb 14, 2016 at 11:59 PM, Jonas Ruesch <jr...@jruesch.ch>
> wrote:
> >
> > > Hi Jens, Randy,
> > >
> > > Thanks for confirming.
> > > Any preference or previous experiences with a zlib-compatible zip
> library
> > > for c#? That one looks promising: http://dotnetzip.codeplex.com/
> > >
> > > Cheers,
> > > Jonas
> > >
> > >
> > > On 13 February 2016 at 00:36, Jens Geyer <je...@hotmail.com>
> wrote:
> > >
> > > > Hi Jonas,
> > > >
> > > > patches and pull requests are welcome.
> > > >
> > > > Because the documentation states 'zlib transport not available for
> > java'
> > > >> (in the meantime it seems to be available though) I had hopes that
> > there
> > > >> is
> > > >> a C# implementation.
> > > >>
> > > >
> > > > That's the problem with logic. It sometimes hits you where you
> > > > expect
> > it
> > > > the least. Just kidding ... ;-) As I said on SO, some things do not
> > exist
> > > > simply because nobody needed them so far. But that does not mean it
> > can't
> > > > or shouldn't be added if someone does need it. Especially TZlib is a
> > good
> > > > thing to have.
> > > >
> > > > Looking forward to your contribution. If you need help with it, just
> > ask,
> > > > and someone will answer.
> > > >
> > > > Have fun,
> > > > JensG
> > > >
> > > >
> > > > -----Ursprüngliche Nachricht----- From: Randy Abernethy
> > > > Sent: Friday, February 12, 2016 3:48 PM
> > > > To: user@thrift.apache.org
> > > > Subject: Re: TZLibTransport for CSharp?
> > > >
> > > >
> > > > Hi Jonas,
> > > >
> > > > The current master does not have a TZlibTransport for C#. Would be a
> > > great
> > > > add though! Patches always welcome.
> > > >
> > > > Best,
> > > > Randy
> > > >
> > > > On Fri, Feb 12, 2016 at 2:46 AM, Jonas Ruesch <jr...@jruesch.ch>
> > > wrote:
> > > >
> > > > Hi,
> > > >>
> > > >> Is there an implementation of the zlib Transport in C# (Thrift
> > 0.9.3)? I
> > > >> looked for it in the lib/csharp source code and believe to have
> > searched
> > > >> the web extensively to no avail.
> > > >>
> > > >> Because the documentation states 'zlib transport not available for
> > java'
> > > >> (in the meantime it seems to be available though) I had hopes that
> > there
> > > >> is
> > > >> a C# implementation.
> > > >>
> > > >> Sorry if this question was asked before (couldn't find any traces).
> > > >>
> > > >> Many thanks for your hints.
> > > >>
> > > >>
> > > >
> > >
> >
>
Re: TZLibTransport for CSharp?
Posted by Randy Abernethy <ra...@gmail.com>.
Hey Jonas,
Thanks for the detailed write up! Will provide great guidance for anyone
interested in implementing TZlibTransport for C#. My experience concurs
with yours, using ZLib for thrift file compression can be useful but it is
not generally an uptick for RPC. Perhaps in that context lack of flush is
acceptable.
-Randy
On Fri, Apr 29, 2016 at 7:16 AM, Jonas Ruesch <jr...@jruesch.ch> wrote:
> Hi Randy, Jens,
>
> Following your suggestions and the existing Java implementation, we had a
> closer look at a possible C# implementation. Below I add a few technical
> notes on what we found. So far we didn't pursue any of the options further,
> because for our specific application we found that in practice the
> uncompressed binary format outperformed the compressed communication in
> terms of time per call (tested with C++ client & server).
>
> Technical notes on possible TZlibTransport implementations in C#/.Net
> (credit: Mario G.):
>
> Dot net supports by default a compression format which is compatible to
> ZLib. It is little effort to create some wrappers to implement the
> TZlibTransport for DotNet. However, the issue is that Thrift is forcing to
> send data by flushing the stream, but the DotNet compressions streams do
> not support flushing. This means that data is only being sent if there is
> enough data available to compress the next block, or if the stream is
> closed.
>
> Thrift only closes the sending stream after the response has been received.
> But as the data has never been sent, data is never received too.
>
> To solve this issue the compression stream can be closed instead of
> flushed. This works as long as it is guaranteed that thrift never sends any
> additional data after calling flush. We didn't investigate if this is
> guaranteed (todo).
>
> If the above condition is not guaranteed, flush support could be
> implemented by using the DotNetZip library which supports the sync_flush
> mode (http://www.bolet.org/~pornin/deflate-flush.html). This mode is only
> supported on a very low level, thus a lot of handling has to be implemented
> manually.
>
> (Sorry, this answer might be better suited for the developer mailing list,
> but I kept it in this thread for better context.)
>
> Best,
> Jonas
>
>
>
>
> On 15 February 2016 at 19:30, Randy Abernethy <ra...@apache.org> wrote:
>
> > Hey Jonas,
> >
> > I think .Net has support for deflate built in, so there's not need for a
> > third party lib. More info here:
> >
> >
> https://msdn.microsoft.com/en-us/library/system.io.compression.deflatestream(v=vs.110).aspx
> >
> > Should be pretty easy to use the Java or C++ TZlibTrans as a model and
> get
> > the same going in C#.
> >
> > Best,
> > Randy
> >
> >
> > On Sun, Feb 14, 2016 at 11:59 PM, Jonas Ruesch <jr...@jruesch.ch>
> wrote:
> >
> > > Hi Jens, Randy,
> > >
> > > Thanks for confirming.
> > > Any preference or previous experiences with a zlib-compatible zip
> library
> > > for c#? That one looks promising: http://dotnetzip.codeplex.com/
> > >
> > > Cheers,
> > > Jonas
> > >
> > >
> > > On 13 February 2016 at 00:36, Jens Geyer <je...@hotmail.com>
> wrote:
> > >
> > > > Hi Jonas,
> > > >
> > > > patches and pull requests are welcome.
> > > >
> > > > Because the documentation states 'zlib transport not available for
> > java'
> > > >> (in the meantime it seems to be available though) I had hopes that
> > there
> > > >> is
> > > >> a C# implementation.
> > > >>
> > > >
> > > > That's the problem with logic. It sometimes hits you where you expect
> > it
> > > > the least. Just kidding ... ;-) As I said on SO, some things do not
> > exist
> > > > simply because nobody needed them so far. But that does not mean it
> > can't
> > > > or shouldn't be added if someone does need it. Especially TZlib is a
> > good
> > > > thing to have.
> > > >
> > > > Looking forward to your contribution. If you need help with it, just
> > ask,
> > > > and someone will answer.
> > > >
> > > > Have fun,
> > > > JensG
> > > >
> > > >
> > > > -----Ursprüngliche Nachricht----- From: Randy Abernethy
> > > > Sent: Friday, February 12, 2016 3:48 PM
> > > > To: user@thrift.apache.org
> > > > Subject: Re: TZLibTransport for CSharp?
> > > >
> > > >
> > > > Hi Jonas,
> > > >
> > > > The current master does not have a TZlibTransport for C#. Would be a
> > > great
> > > > add though! Patches always welcome.
> > > >
> > > > Best,
> > > > Randy
> > > >
> > > > On Fri, Feb 12, 2016 at 2:46 AM, Jonas Ruesch <jr...@jruesch.ch>
> > > wrote:
> > > >
> > > > Hi,
> > > >>
> > > >> Is there an implementation of the zlib Transport in C# (Thrift
> > 0.9.3)? I
> > > >> looked for it in the lib/csharp source code and believe to have
> > searched
> > > >> the web extensively to no avail.
> > > >>
> > > >> Because the documentation states 'zlib transport not available for
> > java'
> > > >> (in the meantime it seems to be available though) I had hopes that
> > there
> > > >> is
> > > >> a C# implementation.
> > > >>
> > > >> Sorry if this question was asked before (couldn't find any traces).
> > > >>
> > > >> Many thanks for your hints.
> > > >>
> > > >>
> > > >
> > >
> >
>