[jira] Updated: (HADOOP-1181) userlogs reader

["stack@archive.org (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack@archive.org updated HADOOP-1181:
--------------------------------------

    Attachment: hadoop1181-v3.patch

I like both of the above suggestions. Arun's integrates the pulling of remote userlogs nicely into the command-line and if Dougs' HADOOP-1199 had been in place, I wouldn't have filed this issue in the first place.

I would suggest though that the attached patch may still have some (minor) merit:  it makes it possible to get a stream on a remote hosts' userlogs dir for grepping, sorting, etc., w/o having to first copy all logs local (or to HDFS) or go via a head node (as per Arun's suggestion) or w/o having to write a mapreduce job as per the Doug suggestion.

Thanks lads

(Redoing patch because v2 had some pollution).

> userlogs reader
> ---------------
>
>                 Key: HADOOP-1181
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1181
>             Project: Hadoop
>          Issue Type: Improvement
>            Reporter: stack@archive.org
>         Attachments: hadoop1181-v2.patch, hadoop1181-v3.patch, hadoop1181.patch
>
>
> My jobs output lots of logging.  I want to be able to quickly parse the logs across the cluster for anomalies.  org.apache.hadoop.tool.Logalyzer looks promising at first but it does not know how to deal with the userlog format  and it wants to first copy all logs local.  Digging, there does not seem to currently be a reader for hadoop userlog format.  TaskLog$Reader is not generally accessible and it too expects logs to be on the local filesystem (The latter is of little good if I want to run the analysis as a mapreduce job).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.