Scribe vs. Chukwa

[Corbin Hoenes <co...@tynt.com>]

Does anyone know of a comparison between Chukwa and Scribe( http://sameerparwani.com/posts/facebook-scribe-server-documentation-and-tutorials )
I am interested in why someone might choose Chukwa vs. Scribe and their pros/cons.

Re: Scribe vs. Chukwa

[Corbin Hoenes <co...@tynt.com>]

Ari,

Thanks for the input.  I found another thread on the general hadoop list that pertains to the the discussion.  I'll post here for future readers:

http://mail-archives.apache.org/mod_mbox/hadoop-general/200911.mbox/%3cC7397F6E.53EA%25eyang@yahoo-inc.com%3e

Your explanation helps a lot though.  I'd second the idea of have a Chukwa-vs page on the website to help people understand what it can do for them.  We are trying to get our logs into HDFS so I think chukwa will provide a lot of value in this regard.

--
Corbin

On Dec 24, 2009, at 9:51 AM, Ariel Rabkin wrote:

> Howdy.
> 
> It's a good question, and maybe we should have a
> Chukwa-vs-the-competition page on the website.
> 
> Chukwa is designed to help collect system logs, and store them in
> HDFS.  Scribe is designed as more of a syslog++.  We went to some
> trouble to make sure our metadata and output formats were MapReduce
> friendly, whereas I think they leave that much more up to you.
> 
> My understanding is that Scribe exposes a Thrift service, only. So if
> you want to use Scribe, you pretty much have to go all-in and modify
> your app and deployment to send logs via Thrift. Chukwa can just read
> whatever files or data sources are lying around.  We can pretty easily
> listen for incoming Thrift RPC calls -- I believe Jerome is currently
> coding that up.
> 
> Chukwa comes with visualization and processing tools, whereas Scribe doesn't.
> 
> Scribe might be slightly lower overhead -- I believe it's C++ and
> Chukwa is Java.  Chukwa isn't too big, either, so the absolute
> difference might not be significant.
> 
> --Ari
> 
> On Wed, Dec 23, 2009 at 3:06 PM, Corbin Hoenes <co...@tynt.com> wrote:
>> Does anyone know of a comparison between Chukwa and Scribe( http://sameerparwani.com/posts/facebook-scribe-server-documentation-and-tutorials )
>> I am interested in why someone might choose Chukwa vs. Scribe and their pros/cons.
>> 
>> 
>> 
> 
> 
> 
> -- 
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department

Corbin Hoenes
corbin@tynt.com
skype: choenes

Re: Scribe vs. Chukwa

[Ariel Rabkin <as...@gmail.com>]

Howdy.

It's a good question, and maybe we should have a
Chukwa-vs-the-competition page on the website.

Chukwa is designed to help collect system logs, and store them in
HDFS.  Scribe is designed as more of a syslog++.  We went to some
trouble to make sure our metadata and output formats were MapReduce
friendly, whereas I think they leave that much more up to you.

My understanding is that Scribe exposes a Thrift service, only. So if
you want to use Scribe, you pretty much have to go all-in and modify
your app and deployment to send logs via Thrift. Chukwa can just read
whatever files or data sources are lying around.  We can pretty easily
listen for incoming Thrift RPC calls -- I believe Jerome is currently
coding that up.

Chukwa comes with visualization and processing tools, whereas Scribe doesn't.

Scribe might be slightly lower overhead -- I believe it's C++ and
Chukwa is Java.  Chukwa isn't too big, either, so the absolute
difference might not be significant.

--Ari

On Wed, Dec 23, 2009 at 3:06 PM, Corbin Hoenes <co...@tynt.com> wrote:
> Does anyone know of a comparison between Chukwa and Scribe( http://sameerparwani.com/posts/facebook-scribe-server-documentation-and-tutorials )
> I am interested in why someone might choose Chukwa vs. Scribe and their pros/cons.
>
>
>



-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department