Using thrift with a message broker. Why?

[Software Dev <st...@gmail.com>]

I've come across some examples of AMQP broker sitting between a client and
service (Thrift). Can someone explain the benefits of this approach as
opposed to just having the client speak directly to the service via the one
of the native thrift transports?

There are two reasons I can think of:

1) Loadbalancing. If we have multiple services as a consumer for a topic,
we essentialy get loadbalancing for free.

2) No need to configure the service endpoints up front. We only need to
write to the broker and not care about which machine, ip, etc can handle
the request.

Neither of which do I find as a compelling reason to add a broker. In use
case 1 we can simply load balance via hardware or HAProxy. Use case two, if
needed, be fixed with some distributed configuration management software
(Zookeeper) where automatic detection is possible.

Can someone please chime in with any input. Thanks

Re: Using thrift with a message broker. Why?

[Software Dev <st...@gmail.com>]

Ok we would be using thrift just for sync RPC at the moment so I think ill
skip the broker in between until its needed. We are actually using Kafka as
a message broker for other long running tasks/event collection so we can
always stick that in between.

Nothing specific here:

https://speakerdeck.com/shenrie/a-lightweight-soa-framework-using-ruby-apache-thrift-and-amqp
http://www.gettingcirrius.com/2011/03/rabbitmq-with-thrift-serialization.html


On Thu, Jan 23, 2014 at 3:06 PM, James Haggerty <mu...@gmail.com> wrote:

> I'm doing this at the moment because it's a better way to do long-running
> async tasks (i.e. using thrift like celery, but never any return values).
> This ends up being more useful than oneway, because if you use oneway you
> don't ack (and there's no simple way to retry if it fails). Oneway also
> ends up 'using up' a process if you're running a Python process pool style
> server, whereas having consumers allows you to constrain how many processes
> are soaked up by the long running function.
>
> At least, it makes sense to me. Do you have links to other people
> describing their use of AMQP?
>
>
> On Fri, Jan 24, 2014 at 5:28 AM, Software Dev <static.void.dev@gmail.com
> >wrote:
>
> > I've come across some examples of AMQP broker sitting between a client
> and
> > service (Thrift). Can someone explain the benefits of this approach as
> > opposed to just having the client speak directly to the service via the
> one
> > of the native thrift transports?
> >
> > There are two reasons I can think of:
> >
> > 1) Loadbalancing. If we have multiple services as a consumer for a topic,
> > we essentialy get loadbalancing for free.
> >
> > 2) No need to configure the service endpoints up front. We only need to
> > write to the broker and not care about which machine, ip, etc can handle
> > the request.
> >
> > Neither of which do I find as a compelling reason to add a broker. In use
> > case 1 we can simply load balance via hardware or HAProxy. Use case two,
> if
> > needed, be fixed with some distributed configuration management software
> > (Zookeeper) where automatic detection is possible.
> >
> > Can someone please chime in with any input. Thanks
> >
>

Re: Using thrift with a message broker. Why?

[James Haggerty <mu...@gmail.com>]

I'm doing this at the moment because it's a better way to do long-running
async tasks (i.e. using thrift like celery, but never any return values).
This ends up being more useful than oneway, because if you use oneway you
don't ack (and there's no simple way to retry if it fails). Oneway also
ends up 'using up' a process if you're running a Python process pool style
server, whereas having consumers allows you to constrain how many processes
are soaked up by the long running function.

At least, it makes sense to me. Do you have links to other people
describing their use of AMQP?


On Fri, Jan 24, 2014 at 5:28 AM, Software Dev <st...@gmail.com>wrote:

> I've come across some examples of AMQP broker sitting between a client and
> service (Thrift). Can someone explain the benefits of this approach as
> opposed to just having the client speak directly to the service via the one
> of the native thrift transports?
>
> There are two reasons I can think of:
>
> 1) Loadbalancing. If we have multiple services as a consumer for a topic,
> we essentialy get loadbalancing for free.
>
> 2) No need to configure the service endpoints up front. We only need to
> write to the broker and not care about which machine, ip, etc can handle
> the request.
>
> Neither of which do I find as a compelling reason to add a broker. In use
> case 1 we can simply load balance via hardware or HAProxy. Use case two, if
> needed, be fixed with some distributed configuration management software
> (Zookeeper) where automatic detection is possible.
>
> Can someone please chime in with any input. Thanks
>

Re: Using thrift with a message broker. Why?

[Randy Abernethy <ra...@gmail.com>]

Hello Blarg,

Not sure if you are referring to folks using the serialization framework
standalone with messaging or if you are referring to folks using the RPC
system with a messaging transport.

One of the great features of Thrift is its IDL driven serialization layer.
A number of folks use Thrift as a cross language serialization framework
for messaging. One of the great things about adopting Thrift serialization
is that you can then use it for both RPC and messaging.

Given that the serialization framework supports plug in protocols, you can
swap JSON for Compact for Binary for Custom easily. You can use JSON for
reach and Bin or Compact for serialization speed.

Being able to describe your interfaces (in this case types which will be
communicated via messaging) once in a crisp IDL for all languages is a
benefit in and of itself.

Some folks use messaging (ZeroMQ in particular) as a transport for Thrift
RPC (maybe what you are seeing). This gives you the benefits of RPC (simple
programming model, etc.) but with many of the benefits of loosely coupled
systems, particularly when you use Thrift async clients. As you mentioned,
this unbinds the client from the server (using the MOM as a stand in). You
can route the RPC traffic to any end point (fail-over), load balance, log
copies of the traffic for replay, etc.

Hope this helps,
Randy






On Thu, Jan 23, 2014 at 10:28 AM, Software Dev <st...@gmail.com>wrote:

> I've come across some examples of AMQP broker sitting between a client and
> service (Thrift). Can someone explain the benefits of this approach as
> opposed to just having the client speak directly to the service via the one
> of the native thrift transports?
>
> There are two reasons I can think of:
>
> 1) Loadbalancing. If we have multiple services as a consumer for a topic,
> we essentialy get loadbalancing for free.
>
> 2) No need to configure the service endpoints up front. We only need to
> write to the broker and not care about which machine, ip, etc can handle
> the request.
>
> Neither of which do I find as a compelling reason to add a broker. In use
> case 1 we can simply load balance via hardware or HAProxy. Use case two, if
> needed, be fixed with some distributed configuration management software
> (Zookeeper) where automatic detection is possible.
>
> Can someone please chime in with any input. Thanks
>