You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by prasenjit mukherjee <pr...@gmail.com> on 2009/09/16 03:01:59 UTC

using hadoop logs to track mapreduce task's progress

I am trying to track my pig job's progress till now via inspecting
hadoop/pig logs.   I see the following set of  logs in my hadoop
logdir :

1. seemingly Hadoop's logs :  datanode, jobtracker, namenode,
secondarynamenode, tasktracker*.log
2. userlogs directory : stderr/stdout/syslog etc.
3. history directory.

>From what it seems ( I may be way off though)  <log-dir>/history
contains some traces of pig specific logs. Any pointer to the format ?
Also any doc/rough sketch of the what these logs say would be of great
help.

-Thanks,
Prasen

Re: using hadoop logs to track mapreduce task's progress

Posted by Jeff Zhang <zj...@gmail.com>.
Why not watching Job Tracker Web interface, it's more intuitive
And Sometimes, one pig script may been converted more than one hadoop jobs.
It's not easy to monitor the progress by seeing the log.
The hadoop logs is for debugging when your job failed.



On Tue, Sep 15, 2009 at 6:01 PM, prasenjit mukherjee
<pr...@gmail.com>wrote:

> I am trying to track my pig job's progress till now via inspecting
> hadoop/pig logs.   I see the following set of  logs in my hadoop
> logdir :
>
> 1. seemingly Hadoop's logs :  datanode, jobtracker, namenode,
> secondarynamenode, tasktracker*.log
> 2. userlogs directory : stderr/stdout/syslog etc.
> 3. history directory.
>
> From what it seems ( I may be way off though)  <log-dir>/history
> contains some traces of pig specific logs. Any pointer to the format ?
> Also any doc/rough sketch of the what these logs say would be of great
> help.
>
> -Thanks,
> Prasen
>

RE: using hadoop logs to track mapreduce task's progress

Posted by zjffdu <zj...@gmail.com>.
Why not watching Job Tracker Web interface, it's more intuitive
And Sometimes, one pig script may been converted more than one hadoop jobs.
It's not easy to monitor the progress by seeing the log.
The hadoop logs is for debugging when your job failed.



-----Original Message-----
From: prasenjit mukherjee [mailto:prasen.bea@gmail.com] 
Sent: 2009年9月15日 18:02
To: pig-user
Subject: using hadoop logs to track mapreduce task's progress

I am trying to track my pig job's progress till now via inspecting
hadoop/pig logs.   I see the following set of  logs in my hadoop
logdir :

1. seemingly Hadoop's logs :  datanode, jobtracker, namenode,
secondarynamenode, tasktracker*.log
2. userlogs directory : stderr/stdout/syslog etc.
3. history directory.

>From what it seems ( I may be way off though)  <log-dir>/history
contains some traces of pig specific logs. Any pointer to the format ?
Also any doc/rough sketch of the what these logs say would be of great
help.

-Thanks,
Prasen