A Big Data Trifecta: Storm, Kafka and Cassandra

[Brian O'Neill <bo...@alumni.brown.edu>]

Philip,

I figured I would reply via blog post. =)
http://brianoneill.blogspot.com/2012/08/a-big-data-trifecta-storm-kafka-and.html

That blog post shows how we pieced together Kafka and Cassandra (via Storm).
With LinkedIn behind Kafka, it is well supported.  They use it in
production. (and most likely we will too =)

Let me know if you end up using it.  Thus far, I think it pairs nicely
with Cassandra, but we don't have it in production yet.

-brian

On Fri, Aug 3, 2012 at 3:41 PM, Milind Parikh <mi...@gmail.com> wrote:
> Kafka is relatively stable and has a active well-supported news-group as
> well.
>
> As discussed by Brian, you would be inverting the paradigm of store-process.
> Essentially in your original approach, you are storing the messages first
> and then processing them after the fact. In the Kafka model, you would
> process the messages as they come in.
>
> Since you are thinking about parallelism anyways, I trust that your
> processing paradigm is inherently paralleizable.
>
> Regards
> Milind
>
>
>
>
>
> On Fri, Aug 3, 2012 at 12:22 PM, Philip Nelson
> <ph...@yahoo.com> wrote:
>>
>> Brian -- thanks.
>>
>> > We were looking to do the same thing, but in the end decided
>> > to go with Kafka.
>> > Given your throughput requirements, Kafka might be a good
>> > option for you as well.
>>
>> This might be off-topic, so I'll keep it short. Kafka is reasonably
>> stable? Mature (I see it's in the Incubator)? Relative to Cassandra?
>>
>> Philip
>>
>>
>



-- 
Brian ONeill
Lead Architect, Health Market Science (http://healthmarketscience.com)
mobile:215.588.6024
blog: http://weblogs.java.net/blog/boneill42/
blog: http://brianoneill.blogspot.com/

Re: A Big Data Trifecta: Storm, Kafka and Cassandra

[Philip Nelson <ph...@yahoo.com>]

--- On Sat, 8/4/12, Brian O'Neill <bo...@alumni.brown.edu> wrote:

> From: Brian O'Neill <bo...@alumni.brown.edu>
> Subject: A Big Data Trifecta: Storm, Kafka and Cassandra
> To: user@cassandra.apache.org
> Date: Saturday, August 4, 2012, 4:41 AM
> Philip,
> 
> I figured I would reply via blog post. =)
> http://brianoneill.blogspot.com/2012/08/a-big-data-trifecta-storm-kafka-and.html

Brian -- thanks again for this. It's always great to get a reference to another approach.