Re: Scribe vs. Chukwa

[Corbin Hoenes <co...@tynt.com>]

Ari,

Thanks for the input.  I found another thread on the general hadoop list that pertains to the the discussion.  I'll post here for future readers:

http://mail-archives.apache.org/mod_mbox/hadoop-general/200911.mbox/%3cC7397F6E.53EA%25eyang@yahoo-inc.com%3e

Your explanation helps a lot though.  I'd second the idea of have a Chukwa-vs page on the website to help people understand what it can do for them.  We are trying to get our logs into HDFS so I think chukwa will provide a lot of value in this regard.

--
Corbin

On Dec 24, 2009, at 9:51 AM, Ariel Rabkin wrote:

> Howdy.
> 
> It's a good question, and maybe we should have a
> Chukwa-vs-the-competition page on the website.
> 
> Chukwa is designed to help collect system logs, and store them in
> HDFS.  Scribe is designed as more of a syslog++.  We went to some
> trouble to make sure our metadata and output formats were MapReduce
> friendly, whereas I think they leave that much more up to you.
> 
> My understanding is that Scribe exposes a Thrift service, only. So if
> you want to use Scribe, you pretty much have to go all-in and modify
> your app and deployment to send logs via Thrift. Chukwa can just read
> whatever files or data sources are lying around.  We can pretty easily
> listen for incoming Thrift RPC calls -- I believe Jerome is currently
> coding that up.
> 
> Chukwa comes with visualization and processing tools, whereas Scribe doesn't.
> 
> Scribe might be slightly lower overhead -- I believe it's C++ and
> Chukwa is Java.  Chukwa isn't too big, either, so the absolute
> difference might not be significant.
> 
> --Ari
> 
> On Wed, Dec 23, 2009 at 3:06 PM, Corbin Hoenes <co...@tynt.com> wrote:
>> Does anyone know of a comparison between Chukwa and Scribe( http://sameerparwani.com/posts/facebook-scribe-server-documentation-and-tutorials )
>> I am interested in why someone might choose Chukwa vs. Scribe and their pros/cons.
>> 
>> 
>> 
> 
> 
> 
> -- 
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department

Corbin Hoenes
corbin@tynt.com
skype: choenes