[jira] Commented: (HADOOP-1181) userlogs reader

["Arun C Murthy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486289 ] 

Arun C Murthy commented on HADOOP-1181:
---------------------------------------

Michael,

  I was going through your patch and conjured up an alternative - this is along the lines of the '-events' sub-command of 'hadoop job'...
  $ hadoop job -events <id>

  How about another command along the same lines:
  $ hadoop job -tasklogs <taskid>  
  i.e.
  $ hadoop job -tasklogs task_0001_m_000009_1
  (or equivalently hadoop job -tasklogs <jobid:taskid:taskattempt> e.g. hadoop job -tasklogs 1:9:1, what do people prefer? I'd vote for the former)

  This can be implemented fairly easily by adding a api for the JobClient to query the JobTracker for the task-tracker on which a particular task-attempt occured, construct the url and then open a http connection to the TaskLog.Reader as is done today in for getting tasklogs from the 

  The advantage is that it would be fairly simple to implement, wouldn't need to make any change to TaskLog.Reader and also would fit along some accepted idioms (e.g. hadoop job -events) ...

 What do you think? Would that satisfy your needs? 
 Others?
  

> userlogs reader
> ---------------
>
>                 Key: HADOOP-1181
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1181
>             Project: Hadoop
>          Issue Type: Improvement
>            Reporter: stack@archive.org
>         Attachments: hadoop1181-v2.patch, hadoop1181.patch
>
>
> My jobs output lots of logging.  I want to be able to quickly parse the logs across the cluster for anomalies.  org.apache.hadoop.tool.Logalyzer looks promising at first but it does not know how to deal with the userlog format  and it wants to first copy all logs local.  Digging, there does not seem to currently be a reader for hadoop userlog format.  TaskLog$Reader is not generally accessible and it too expects logs to be on the local filesystem (The latter is of little good if I want to run the analysis as a mapreduce job).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.