Remote consumer

[Muhzin <rm...@gmail.com>]

Hi,

We are architecting an  application in AWS with  multi region deployment
 and were  evaluating kafka as the messaging system for  data replication
and activity tracking.

The initial idea is to have the producer and broker cluster to reside in
Ireland. We wanted to know the feasibility of having a remote consumer in
Singapore.
Found a link to improve throughput for a remote consumer -
http://bit.ly/1cp7T6Z
#) Has anybody done something similar - if so, could you please shed some
light on how latency will play and some of the best practices that could be
done.
#) What would be a good way to secure the communication if it is happening
over the internet. We were thinking about SSL tunnel (using OpenVPN) as
described in  http://aws.amazon.com/articles/0639686206802544. Thoughts?

-- 
BR
Muhsin

Re: Remote consumer

[Joe Stein <jo...@stealth.ly>]

One option would be to use Cassandra for the multi data center replication
and have Kafka Consumes update the Cassandra ring in each data center.

This allows you to have active / active / active applications.  It also
allows you to choose a ring and map/reduce the data out of it (like with
Pig) which will saturate that ring so you want to-do it in an availability
zone where your application is not serving out of.

Every data center has its own "commit log" thanks to the Kafka broker
cluster and data center replication from a database with a Cassandra ring.

This is the setup I have used before and continue to work on now.

I have been meaning to start open sourcing some of this (it uses Thrift
also).


On Thu, Oct 31, 2013 at 8:37 AM, Muhzin <rm...@gmail.com> wrote:

> Hi,
>
> We are architecting an  application in AWS with  multi region deployment
>  and were  evaluating kafka as the messaging system for  data replication
> and activity tracking.
>
> The initial idea is to have the producer and broker cluster to reside in
> Ireland. We wanted to know the feasibility of having a remote consumer in
> Singapore.
> Found a link to improve throughput for a remote consumer -
> http://bit.ly/1cp7T6Z
> #) Has anybody done something similar - if so, could you please shed some
> light on how latency will play and some of the best practices that could be
> done.
> #) What would be a good way to secure the communication if it is happening
> over the internet. We were thinking about SSL tunnel (using OpenVPN) as
> described in  http://aws.amazon.com/articles/0639686206802544. Thoughts?
>
> --
> BR
> Muhsin
>