You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Hank Cohen <ha...@altior.com> on 2012/08/17 06:29:39 UTC
Log file parsing
Are there any utilities available to help parse jobtracker log files?
Hank Cohen
hank.cohen@altior.com<ma...@altior.com>
www.altior.com<http://www.altior.com>
Re: Log file parsing
Posted by Prashant Kommireddi <pr...@gmail.com>.
Take a look at Pig's HadoopJobHistoryLoader
http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/piggybank/storage/HadoopJobHistoryLoader.html
On Thu, Aug 16, 2012 at 9:34 PM, peter <zh...@gmail.com> wrote:
> Now, no utilities for job tracker log .
>
>
> --
> peter
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> On 2012年8月17日Friday at 下午12:29, Hank Cohen wrote:
>
> Are there any utilities available to help parse jobtracker log files?****
>
> ** **
>
> *Hank Cohen*****
>
> ** **
>
> hank.cohen@altior.com****
>
> www.altior.com****
>
> ** **
>
> ** **
>
> ** **
>
>
>
Re: Log file parsing
Posted by Prashant Kommireddi <pr...@gmail.com>.
Take a look at Pig's HadoopJobHistoryLoader
http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/piggybank/storage/HadoopJobHistoryLoader.html
On Thu, Aug 16, 2012 at 9:34 PM, peter <zh...@gmail.com> wrote:
> Now, no utilities for job tracker log .
>
>
> --
> peter
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> On 2012年8月17日Friday at 下午12:29, Hank Cohen wrote:
>
> Are there any utilities available to help parse jobtracker log files?****
>
> ** **
>
> *Hank Cohen*****
>
> ** **
>
> hank.cohen@altior.com****
>
> www.altior.com****
>
> ** **
>
> ** **
>
> ** **
>
>
>
Re: Log file parsing
Posted by Prashant Kommireddi <pr...@gmail.com>.
Take a look at Pig's HadoopJobHistoryLoader
http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/piggybank/storage/HadoopJobHistoryLoader.html
On Thu, Aug 16, 2012 at 9:34 PM, peter <zh...@gmail.com> wrote:
> Now, no utilities for job tracker log .
>
>
> --
> peter
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> On 2012年8月17日Friday at 下午12:29, Hank Cohen wrote:
>
> Are there any utilities available to help parse jobtracker log files?****
>
> ** **
>
> *Hank Cohen*****
>
> ** **
>
> hank.cohen@altior.com****
>
> www.altior.com****
>
> ** **
>
> ** **
>
> ** **
>
>
>
Re: Log file parsing
Posted by Prashant Kommireddi <pr...@gmail.com>.
Take a look at Pig's HadoopJobHistoryLoader
http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/piggybank/storage/HadoopJobHistoryLoader.html
On Thu, Aug 16, 2012 at 9:34 PM, peter <zh...@gmail.com> wrote:
> Now, no utilities for job tracker log .
>
>
> --
> peter
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> On 2012年8月17日Friday at 下午12:29, Hank Cohen wrote:
>
> Are there any utilities available to help parse jobtracker log files?****
>
> ** **
>
> *Hank Cohen*****
>
> ** **
>
> hank.cohen@altior.com****
>
> www.altior.com****
>
> ** **
>
> ** **
>
> ** **
>
>
>
Re: Log file parsing
Posted by peter <zh...@gmail.com>.
Now, no utilities for job tracker log .
--
peter
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On 2012年8月17日Friday at 下午12:29, Hank Cohen wrote:
> Are there any utilities available to help parse jobtracker log files?
>
> Hank Cohen
>
> hank.cohen@altior.com (mailto:hank.cohen@altior.com)
> www.altior.com (http://www.altior.com)
>
>
>
>
>
>
RE: Log file parsing
Posted by Hank Cohen <ha...@altior.com>.
Raj,
Thank you! This is perfect.
I was about to learn Perl so I could write a similar program. I don't think awk or sed would cut it.
Now I don't have to and I can get on to my analysis right away.
Hank Cohen
From: Raj Vishwanathan [mailto:rajvish@yahoo.com]
Sent: Friday, August 17, 2012 3:34 PM
To: user@hadoop.apache.org; stan.rosenberg@gmail.com
Subject: Re: Log file parsing
if you want to parse job log files, you can take a look at
https://github.com/rajvish/hadoop-summary
Raj
________________________________
From: Stan Rosenberg <st...@gmail.com>>
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Sent: Friday, August 17, 2012 9:59 AM
Subject: Re: Log file parsing
org.apache.hadoop.mapred.DefaultJobHistoryParser is a good place to start.
On Fri, Aug 17, 2012 at 12:29 AM, Hank Cohen <ha...@altior.com>> wrote:
> Are there any utilities available to help parse jobtracker log files?
>
>
>
> Hank Cohen
>
>
>
> hank.cohen@altior.com<ma...@altior.com>
>
> www.altior.com<http://www.altior.com>
>
>
>
>
>
>
RE: Log file parsing
Posted by Hank Cohen <ha...@altior.com>.
Raj,
Thank you! This is perfect.
I was about to learn Perl so I could write a similar program. I don't think awk or sed would cut it.
Now I don't have to and I can get on to my analysis right away.
Hank Cohen
From: Raj Vishwanathan [mailto:rajvish@yahoo.com]
Sent: Friday, August 17, 2012 3:34 PM
To: user@hadoop.apache.org; stan.rosenberg@gmail.com
Subject: Re: Log file parsing
if you want to parse job log files, you can take a look at
https://github.com/rajvish/hadoop-summary
Raj
________________________________
From: Stan Rosenberg <st...@gmail.com>>
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Sent: Friday, August 17, 2012 9:59 AM
Subject: Re: Log file parsing
org.apache.hadoop.mapred.DefaultJobHistoryParser is a good place to start.
On Fri, Aug 17, 2012 at 12:29 AM, Hank Cohen <ha...@altior.com>> wrote:
> Are there any utilities available to help parse jobtracker log files?
>
>
>
> Hank Cohen
>
>
>
> hank.cohen@altior.com<ma...@altior.com>
>
> www.altior.com<http://www.altior.com>
>
>
>
>
>
>
RE: Log file parsing
Posted by Hank Cohen <ha...@altior.com>.
Raj,
Thank you! This is perfect.
I was about to learn Perl so I could write a similar program. I don't think awk or sed would cut it.
Now I don't have to and I can get on to my analysis right away.
Hank Cohen
From: Raj Vishwanathan [mailto:rajvish@yahoo.com]
Sent: Friday, August 17, 2012 3:34 PM
To: user@hadoop.apache.org; stan.rosenberg@gmail.com
Subject: Re: Log file parsing
if you want to parse job log files, you can take a look at
https://github.com/rajvish/hadoop-summary
Raj
________________________________
From: Stan Rosenberg <st...@gmail.com>>
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Sent: Friday, August 17, 2012 9:59 AM
Subject: Re: Log file parsing
org.apache.hadoop.mapred.DefaultJobHistoryParser is a good place to start.
On Fri, Aug 17, 2012 at 12:29 AM, Hank Cohen <ha...@altior.com>> wrote:
> Are there any utilities available to help parse jobtracker log files?
>
>
>
> Hank Cohen
>
>
>
> hank.cohen@altior.com<ma...@altior.com>
>
> www.altior.com<http://www.altior.com>
>
>
>
>
>
>
RE: Log file parsing
Posted by Hank Cohen <ha...@altior.com>.
Raj,
Thank you! This is perfect.
I was about to learn Perl so I could write a similar program. I don't think awk or sed would cut it.
Now I don't have to and I can get on to my analysis right away.
Hank Cohen
From: Raj Vishwanathan [mailto:rajvish@yahoo.com]
Sent: Friday, August 17, 2012 3:34 PM
To: user@hadoop.apache.org; stan.rosenberg@gmail.com
Subject: Re: Log file parsing
if you want to parse job log files, you can take a look at
https://github.com/rajvish/hadoop-summary
Raj
________________________________
From: Stan Rosenberg <st...@gmail.com>>
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Sent: Friday, August 17, 2012 9:59 AM
Subject: Re: Log file parsing
org.apache.hadoop.mapred.DefaultJobHistoryParser is a good place to start.
On Fri, Aug 17, 2012 at 12:29 AM, Hank Cohen <ha...@altior.com>> wrote:
> Are there any utilities available to help parse jobtracker log files?
>
>
>
> Hank Cohen
>
>
>
> hank.cohen@altior.com<ma...@altior.com>
>
> www.altior.com<http://www.altior.com>
>
>
>
>
>
>
Re: Log file parsing
Posted by Raj Vishwanathan <ra...@yahoo.com>.
if you want to parse job log files, you can take a look at
https://github.com/rajvish/hadoop-summary
Raj
>________________________________
> From: Stan Rosenberg <st...@gmail.com>
>To: user@hadoop.apache.org
>Sent: Friday, August 17, 2012 9:59 AM
>Subject: Re: Log file parsing
>
>org.apache.hadoop.mapred.DefaultJobHistoryParser is a good place to start.
>
>On Fri, Aug 17, 2012 at 12:29 AM, Hank Cohen <ha...@altior.com> wrote:
>> Are there any utilities available to help parse jobtracker log files?
>>
>>
>>
>> Hank Cohen
>>
>>
>>
>> hank.cohen@altior.com
>>
>> www.altior.com
>>
>>
>>
>>
>>
>>
>
>
>
Re: Log file parsing
Posted by Raj Vishwanathan <ra...@yahoo.com>.
if you want to parse job log files, you can take a look at
https://github.com/rajvish/hadoop-summary
Raj
>________________________________
> From: Stan Rosenberg <st...@gmail.com>
>To: user@hadoop.apache.org
>Sent: Friday, August 17, 2012 9:59 AM
>Subject: Re: Log file parsing
>
>org.apache.hadoop.mapred.DefaultJobHistoryParser is a good place to start.
>
>On Fri, Aug 17, 2012 at 12:29 AM, Hank Cohen <ha...@altior.com> wrote:
>> Are there any utilities available to help parse jobtracker log files?
>>
>>
>>
>> Hank Cohen
>>
>>
>>
>> hank.cohen@altior.com
>>
>> www.altior.com
>>
>>
>>
>>
>>
>>
>
>
>
Re: Log file parsing
Posted by Raj Vishwanathan <ra...@yahoo.com>.
if you want to parse job log files, you can take a look at
https://github.com/rajvish/hadoop-summary
Raj
>________________________________
> From: Stan Rosenberg <st...@gmail.com>
>To: user@hadoop.apache.org
>Sent: Friday, August 17, 2012 9:59 AM
>Subject: Re: Log file parsing
>
>org.apache.hadoop.mapred.DefaultJobHistoryParser is a good place to start.
>
>On Fri, Aug 17, 2012 at 12:29 AM, Hank Cohen <ha...@altior.com> wrote:
>> Are there any utilities available to help parse jobtracker log files?
>>
>>
>>
>> Hank Cohen
>>
>>
>>
>> hank.cohen@altior.com
>>
>> www.altior.com
>>
>>
>>
>>
>>
>>
>
>
>
Re: Log file parsing
Posted by Raj Vishwanathan <ra...@yahoo.com>.
if you want to parse job log files, you can take a look at
https://github.com/rajvish/hadoop-summary
Raj
>________________________________
> From: Stan Rosenberg <st...@gmail.com>
>To: user@hadoop.apache.org
>Sent: Friday, August 17, 2012 9:59 AM
>Subject: Re: Log file parsing
>
>org.apache.hadoop.mapred.DefaultJobHistoryParser is a good place to start.
>
>On Fri, Aug 17, 2012 at 12:29 AM, Hank Cohen <ha...@altior.com> wrote:
>> Are there any utilities available to help parse jobtracker log files?
>>
>>
>>
>> Hank Cohen
>>
>>
>>
>> hank.cohen@altior.com
>>
>> www.altior.com
>>
>>
>>
>>
>>
>>
>
>
>
Re: Log file parsing
Posted by Stan Rosenberg <st...@gmail.com>.
org.apache.hadoop.mapred.DefaultJobHistoryParser is a good place to start.
On Fri, Aug 17, 2012 at 12:29 AM, Hank Cohen <ha...@altior.com> wrote:
> Are there any utilities available to help parse jobtracker log files?
>
>
>
> Hank Cohen
>
>
>
> hank.cohen@altior.com
>
> www.altior.com
>
>
>
>
>
>
Re: Log file parsing
Posted by Stan Rosenberg <st...@gmail.com>.
org.apache.hadoop.mapred.DefaultJobHistoryParser is a good place to start.
On Fri, Aug 17, 2012 at 12:29 AM, Hank Cohen <ha...@altior.com> wrote:
> Are there any utilities available to help parse jobtracker log files?
>
>
>
> Hank Cohen
>
>
>
> hank.cohen@altior.com
>
> www.altior.com
>
>
>
>
>
>
Re: Log file parsing
Posted by peter <zh...@gmail.com>.
Now, no utilities for job tracker log .
--
peter
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On 2012年8月17日Friday at 下午12:29, Hank Cohen wrote:
> Are there any utilities available to help parse jobtracker log files?
>
> Hank Cohen
>
> hank.cohen@altior.com (mailto:hank.cohen@altior.com)
> www.altior.com (http://www.altior.com)
>
>
>
>
>
>
Re: Log file parsing
Posted by Ariel Marcus <ar...@arielmarcus.com>.
Hi,
I would also look into logstash if you are looking for analyzing logs
across your cluster:
http://logstash.net/docs/1.1.1/
I cannot recommend it from experience, but it seems to be a tool built for
this type of problem.
Best,
Ariel
On Aug 17, 2012 12:52 AM, "Harsh J" <ha...@cloudera.com> wrote:
> Hey Hank,
>
> What are you looking to extract out of this service log exactly?
>
> On Fri, Aug 17, 2012 at 9:59 AM, Hank Cohen <ha...@altior.com> wrote:
> > Are there any utilities available to help parse jobtracker log files?
> >
> >
> >
> > Hank Cohen
> >
> >
> >
> > hank.cohen@altior.com
> >
> > www.altior.com
> >
> >
> >
> >
> >
> >
>
>
>
> --
> Harsh J
>
Re: Log file parsing
Posted by Ariel Marcus <ar...@arielmarcus.com>.
Hi,
I would also look into logstash if you are looking for analyzing logs
across your cluster:
http://logstash.net/docs/1.1.1/
I cannot recommend it from experience, but it seems to be a tool built for
this type of problem.
Best,
Ariel
On Aug 17, 2012 12:52 AM, "Harsh J" <ha...@cloudera.com> wrote:
> Hey Hank,
>
> What are you looking to extract out of this service log exactly?
>
> On Fri, Aug 17, 2012 at 9:59 AM, Hank Cohen <ha...@altior.com> wrote:
> > Are there any utilities available to help parse jobtracker log files?
> >
> >
> >
> > Hank Cohen
> >
> >
> >
> > hank.cohen@altior.com
> >
> > www.altior.com
> >
> >
> >
> >
> >
> >
>
>
>
> --
> Harsh J
>
Re: Log file parsing
Posted by Ariel Marcus <ar...@arielmarcus.com>.
Hi,
I would also look into logstash if you are looking for analyzing logs
across your cluster:
http://logstash.net/docs/1.1.1/
I cannot recommend it from experience, but it seems to be a tool built for
this type of problem.
Best,
Ariel
On Aug 17, 2012 12:52 AM, "Harsh J" <ha...@cloudera.com> wrote:
> Hey Hank,
>
> What are you looking to extract out of this service log exactly?
>
> On Fri, Aug 17, 2012 at 9:59 AM, Hank Cohen <ha...@altior.com> wrote:
> > Are there any utilities available to help parse jobtracker log files?
> >
> >
> >
> > Hank Cohen
> >
> >
> >
> > hank.cohen@altior.com
> >
> > www.altior.com
> >
> >
> >
> >
> >
> >
>
>
>
> --
> Harsh J
>
Re: Log file parsing
Posted by Ariel Marcus <ar...@arielmarcus.com>.
Hi,
I would also look into logstash if you are looking for analyzing logs
across your cluster:
http://logstash.net/docs/1.1.1/
I cannot recommend it from experience, but it seems to be a tool built for
this type of problem.
Best,
Ariel
On Aug 17, 2012 12:52 AM, "Harsh J" <ha...@cloudera.com> wrote:
> Hey Hank,
>
> What are you looking to extract out of this service log exactly?
>
> On Fri, Aug 17, 2012 at 9:59 AM, Hank Cohen <ha...@altior.com> wrote:
> > Are there any utilities available to help parse jobtracker log files?
> >
> >
> >
> > Hank Cohen
> >
> >
> >
> > hank.cohen@altior.com
> >
> > www.altior.com
> >
> >
> >
> >
> >
> >
>
>
>
> --
> Harsh J
>
Re: Log file parsing
Posted by Harsh J <ha...@cloudera.com>.
Hey Hank,
What are you looking to extract out of this service log exactly?
On Fri, Aug 17, 2012 at 9:59 AM, Hank Cohen <ha...@altior.com> wrote:
> Are there any utilities available to help parse jobtracker log files?
>
>
>
> Hank Cohen
>
>
>
> hank.cohen@altior.com
>
> www.altior.com
>
>
>
>
>
>
--
Harsh J
Re: Log file parsing
Posted by Harsh J <ha...@cloudera.com>.
Hey Hank,
What are you looking to extract out of this service log exactly?
On Fri, Aug 17, 2012 at 9:59 AM, Hank Cohen <ha...@altior.com> wrote:
> Are there any utilities available to help parse jobtracker log files?
>
>
>
> Hank Cohen
>
>
>
> hank.cohen@altior.com
>
> www.altior.com
>
>
>
>
>
>
--
Harsh J
Re: Log file parsing
Posted by Harsh J <ha...@cloudera.com>.
Hey Hank,
What are you looking to extract out of this service log exactly?
On Fri, Aug 17, 2012 at 9:59 AM, Hank Cohen <ha...@altior.com> wrote:
> Are there any utilities available to help parse jobtracker log files?
>
>
>
> Hank Cohen
>
>
>
> hank.cohen@altior.com
>
> www.altior.com
>
>
>
>
>
>
--
Harsh J
Re: Log file parsing
Posted by Stan Rosenberg <st...@gmail.com>.
org.apache.hadoop.mapred.DefaultJobHistoryParser is a good place to start.
On Fri, Aug 17, 2012 at 12:29 AM, Hank Cohen <ha...@altior.com> wrote:
> Are there any utilities available to help parse jobtracker log files?
>
>
>
> Hank Cohen
>
>
>
> hank.cohen@altior.com
>
> www.altior.com
>
>
>
>
>
>
Re: Log file parsing
Posted by peter <zh...@gmail.com>.
Now, no utilities for job tracker log .
--
peter
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On 2012年8月17日Friday at 下午12:29, Hank Cohen wrote:
> Are there any utilities available to help parse jobtracker log files?
>
> Hank Cohen
>
> hank.cohen@altior.com (mailto:hank.cohen@altior.com)
> www.altior.com (http://www.altior.com)
>
>
>
>
>
>
Re: Log file parsing
Posted by Stan Rosenberg <st...@gmail.com>.
org.apache.hadoop.mapred.DefaultJobHistoryParser is a good place to start.
On Fri, Aug 17, 2012 at 12:29 AM, Hank Cohen <ha...@altior.com> wrote:
> Are there any utilities available to help parse jobtracker log files?
>
>
>
> Hank Cohen
>
>
>
> hank.cohen@altior.com
>
> www.altior.com
>
>
>
>
>
>
Re: Log file parsing
Posted by peter <zh...@gmail.com>.
Now, no utilities for job tracker log .
--
peter
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On 2012年8月17日Friday at 下午12:29, Hank Cohen wrote:
> Are there any utilities available to help parse jobtracker log files?
>
> Hank Cohen
>
> hank.cohen@altior.com (mailto:hank.cohen@altior.com)
> www.altior.com (http://www.altior.com)
>
>
>
>
>
>
Re: Log file parsing
Posted by Harsh J <ha...@cloudera.com>.
Hey Hank,
What are you looking to extract out of this service log exactly?
On Fri, Aug 17, 2012 at 9:59 AM, Hank Cohen <ha...@altior.com> wrote:
> Are there any utilities available to help parse jobtracker log files?
>
>
>
> Hank Cohen
>
>
>
> hank.cohen@altior.com
>
> www.altior.com
>
>
>
>
>
>
--
Harsh J