Considering Chukwa for Log Processing

[msacks <nt...@gmail.com>]

Hello,
I am considering using Chukwa for an open source method for processing
 and analyzing application and sys logs (top problems, historical
error analysis). Would Chukwa be a good solution for this purpose, or
is it overkill?

Thanks,
msacks

Re: Considering Chukwa for Log Processing

[Eric Yang <er...@gmail.com>]

There is currently no search function implemented in HICC.  You will need to write your own script to do search.  Pig script grep is probably easiest to write.
Hope this helps.

regards,
Eric

On Aug 1, 2011, at 1:46 PM, AD wrote:

> I was wondering what is the best way to do log search after the aggregation
> step. Are there ways to do searches from the HICC? From what I understood is
> that I need to write my own PIG scripts. Are they good enough if I want to
> grep for specific errors and keywords from HBase?
> 
> Thanks,
> Abhijit
> 
> --
> View this message in context: http://apache-chukwa.679492.n3.nabble.com/Considering-Chukwa-for-Log-Processing-tp2645288p3217015.html
> Sent from the Chukwa - Users mailing list archive at Nabble.com.

Re: Considering Chukwa for Log Processing

[AD <ab...@gmail.com>]

I was wondering what is the best way to do log search after the aggregation
step. Are there ways to do searches from the HICC? From what I understood is
that I need to write my own PIG scripts. Are they good enough if I want to
grep for specific errors and keywords from HBase?

Thanks,
Abhijit

--
View this message in context: http://apache-chukwa.679492.n3.nabble.com/Considering-Chukwa-for-Log-Processing-tp2645288p3217015.html
Sent from the Chukwa - Users mailing list archive at Nabble.com.

Re: Considering Chukwa for Log Processing

[Eric Yang <ey...@yahoo-inc.com>]

Hi,

Chukwa is a generic framework that some people have adopt to use this framework for data collection and data analysis.  Chukwa's strength is to prepare data into map/reduce friendly format and size.   The current implementation is good for historical trend analysis, but it is not optimized for log search.  Chukwa shines when there are thousands of machines generating logs rapidly.  There are 3 tasks to get Chukwa to process new type of log data.

1. Specify which adaptor to stream the log file.
2. Define a demux parser for the new log file.
3. Write some data analytics script in mapreduce or pig script (compute aggregates).

Hope this helps.

Regards,
Eric

On 3/6/11 11:29 PM, "msacks" <nt...@gmail.com> wrote:

Hello,
I am considering using Chukwa for an open source method for processing
 and analyzing application and sys logs (top problems, historical
error analysis). Would Chukwa be a good solution for this purpose, or
is it overkill?

Thanks,
msacks