You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by Wilm Schumacher <wi...@cawoom.com> on 2014/11/12 10:02:57 UTC
sending large byte objects via thrift
Hi,
I always read that thrift is not optimized for sending large byte
objects (e.g. files) via thrift and that e.g. http should be used instead.
However, I want to discuss the use case of connecting a frontend (e.g. a
web application) with a backend. This means that I have a handful of
connections per thrift server and both client and server are in the same
network, perhaps even the same rack. Thus network is only limited by the
network card chip
In this case I could send a byte object via thrift or create a e.g. http
server to recieve and send large obejctes via REST api.
But in this use case I do not understand why I shouldn't use thrift.
Whether I block a thread for a "http get" or a "thrift send" doesn't
make a difference, isn't it?
The advantage of the "thrift only" solution would be the simplicity. I
would need only one server, and one client per application server. Thus,
less code, less errors, less security issues etc..
The disadvantage of the "thrift only" solution would be the lack of
fancy http features, like compression. But compression wouldn't be a
good plan in this use case anyway, because bandwidth isn't the
bottleneck in this application of thrift and compression would block a
lot CPU.
Do I miss something? Is it perhaps the memory consumption of thrift for
large messages?
Best wishes
Wilm
Re: sending large byte objects via thrift
Posted by Wilm Schumacher <wi...@cawoom.com>.
So, the memory consumption it is. Thx for the fast reply.
Best wishes,
Wilm
Am 12.11.2014 um 10:11 schrieb Bryan Pham:
> Hi Will,
>
> Typically web server are better at handling chunk or stream byte streams
> using disk as a buffer. Thrift is memory only, so you can't expect to
> scale well for payload of 1GB or 10GB (large file uploads.) Typically, you
> want the front-end server to handle streaming body and use binary data type
> of maximum 1MB as your back-end infrastructure.
>
> Best,
> Bryan
>
> On Wed, Nov 12, 2014 at 1:02 AM, Wilm Schumacher <wilm.schumacher@cawoom.com
>> wrote:
>
>> Hi,
>>
>> I always read that thrift is not optimized for sending large byte
>> objects (e.g. files) via thrift and that e.g. http should be used instead.
>>
>> However, I want to discuss the use case of connecting a frontend (e.g. a
>> web application) with a backend. This means that I have a handful of
>> connections per thrift server and both client and server are in the same
>> network, perhaps even the same rack. Thus network is only limited by the
>> network card chip
>>
>> In this case I could send a byte object via thrift or create a e.g. http
>> server to recieve and send large obejctes via REST api.
>>
>> But in this use case I do not understand why I shouldn't use thrift.
>> Whether I block a thread for a "http get" or a "thrift send" doesn't
>> make a difference, isn't it?
>>
>> The advantage of the "thrift only" solution would be the simplicity. I
>> would need only one server, and one client per application server. Thus,
>> less code, less errors, less security issues etc..
>>
>> The disadvantage of the "thrift only" solution would be the lack of
>> fancy http features, like compression. But compression wouldn't be a
>> good plan in this use case anyway, because bandwidth isn't the
>> bottleneck in this application of thrift and compression would block a
>> lot CPU.
>>
>> Do I miss something? Is it perhaps the memory consumption of thrift for
>> large messages?
>>
>> Best wishes
>>
>> Wilm
>>
>
>
>
Re: sending large byte objects via thrift
Posted by Bryan Pham <br...@cloudtenna.com>.
Hi Will,
Typically web server are better at handling chunk or stream byte streams
using disk as a buffer. Thrift is memory only, so you can't expect to
scale well for payload of 1GB or 10GB (large file uploads.) Typically, you
want the front-end server to handle streaming body and use binary data type
of maximum 1MB as your back-end infrastructure.
Best,
Bryan
On Wed, Nov 12, 2014 at 1:02 AM, Wilm Schumacher <wilm.schumacher@cawoom.com
> wrote:
> Hi,
>
> I always read that thrift is not optimized for sending large byte
> objects (e.g. files) via thrift and that e.g. http should be used instead.
>
> However, I want to discuss the use case of connecting a frontend (e.g. a
> web application) with a backend. This means that I have a handful of
> connections per thrift server and both client and server are in the same
> network, perhaps even the same rack. Thus network is only limited by the
> network card chip
>
> In this case I could send a byte object via thrift or create a e.g. http
> server to recieve and send large obejctes via REST api.
>
> But in this use case I do not understand why I shouldn't use thrift.
> Whether I block a thread for a "http get" or a "thrift send" doesn't
> make a difference, isn't it?
>
> The advantage of the "thrift only" solution would be the simplicity. I
> would need only one server, and one client per application server. Thus,
> less code, less errors, less security issues etc..
>
> The disadvantage of the "thrift only" solution would be the lack of
> fancy http features, like compression. But compression wouldn't be a
> good plan in this use case anyway, because bandwidth isn't the
> bottleneck in this application of thrift and compression would block a
> lot CPU.
>
> Do I miss something? Is it perhaps the memory consumption of thrift for
> large messages?
>
> Best wishes
>
> Wilm
>
--
Best,
Bryan Pham
Founder, CloudTenna Inc.