You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "stack@archive.org (JIRA)" <ji...@apache.org> on 2007/03/29 00:41:25 UTC
[jira] Created: (HADOOP-1181) userlogs reader
userlogs reader
---------------
Key: HADOOP-1181
URL: https://issues.apache.org/jira/browse/HADOOP-1181
Project: Hadoop
Issue Type: Improvement
Reporter: stack@archive.org
My jobs output lots of logging. I want to be able to quickly parse the logs across the cluster for anomalies. org.apache.hadoop.tool.Logalyzer looks promising at first but it does not know how to deal with the userlog format and it wants to first copy all logs local. Digging, there does not seem to currently be a reader for hadoop userlog format. TaskLog$Reader is not generally accessible and it too expects logs to be on the local filesystem (The latter is of little good if I want to run the analysis as a mapreduce job).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1181) userlogs reader
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486407 ]
Doug Cutting commented on HADOOP-1181:
--------------------------------------
Related to this, it would be great to have an InputFormat implementation that includes all the task logs from a job. Folks should be able to do something like:
job = new JobConf();
job.setInputFormatClass(TaskLogInputFormat.class);
TaskLogInputFormat.setJobId(jobId);
...
I mentioned this before, in HADOOP-342. Examining logs should be as lightweight as possible, to facilitate debugging. It should not require a copy to HDFS. A faster debug loop is like a faster search engine: it makes people more productive. The sooner one can find that, e.g., most tasks failed with a NullPointerException on line 723, the better.
Should I file a separate issue for this?
> userlogs reader
> ---------------
>
> Key: HADOOP-1181
> URL: https://issues.apache.org/jira/browse/HADOOP-1181
> Project: Hadoop
> Issue Type: Improvement
> Reporter: stack@archive.org
> Attachments: hadoop1181-v2.patch, hadoop1181.patch
>
>
> My jobs output lots of logging. I want to be able to quickly parse the logs across the cluster for anomalies. org.apache.hadoop.tool.Logalyzer looks promising at first but it does not know how to deal with the userlog format and it wants to first copy all logs local. Digging, there does not seem to currently be a reader for hadoop userlog format. TaskLog$Reader is not generally accessible and it too expects logs to be on the local filesystem (The latter is of little good if I want to run the analysis as a mapreduce job).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1181) userlogs reader
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486435 ]
Arun C Murthy commented on HADOOP-1181:
---------------------------------------
Actually (we probably should take this discussion elsewhere, so +1 for a separate issue), the idea of starting another mapred job to get the logs of a previous mapred job doesn't seem right ...
How about extending my previous idea to get logs of all tasks via tasklog.jsp like so:
Get all tasklogs for a given jobid
$ hadoop job <id> -tasklogs
Get all tasklogs for all maps/reduces
$ hadoop job <id> -tasklogs maps
$ hadoop job <id> -tasklogs reduces
(extend this for succesfull/failed maps/reduces?)
Get tasklog for given <jobid,taskid>
$ hadoop job <id> -tasklogs <taskid>
Thoughts?
> userlogs reader
> ---------------
>
> Key: HADOOP-1181
> URL: https://issues.apache.org/jira/browse/HADOOP-1181
> Project: Hadoop
> Issue Type: Improvement
> Reporter: stack@archive.org
> Attachments: hadoop1181-v2.patch, hadoop1181.patch
>
>
> My jobs output lots of logging. I want to be able to quickly parse the logs across the cluster for anomalies. org.apache.hadoop.tool.Logalyzer looks promising at first but it does not know how to deal with the userlog format and it wants to first copy all logs local. Digging, there does not seem to currently be a reader for hadoop userlog format. TaskLog$Reader is not generally accessible and it too expects logs to be on the local filesystem (The latter is of little good if I want to run the analysis as a mapreduce job).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1181) userlogs reader
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486289 ]
Arun C Murthy commented on HADOOP-1181:
---------------------------------------
Michael,
I was going through your patch and conjured up an alternative - this is along the lines of the '-events' sub-command of 'hadoop job'...
$ hadoop job -events <id>
How about another command along the same lines:
$ hadoop job -tasklogs <taskid>
i.e.
$ hadoop job -tasklogs task_0001_m_000009_1
(or equivalently hadoop job -tasklogs <jobid:taskid:taskattempt> e.g. hadoop job -tasklogs 1:9:1, what do people prefer? I'd vote for the former)
This can be implemented fairly easily by adding a api for the JobClient to query the JobTracker for the task-tracker on which a particular task-attempt occured, construct the url and then open a http connection to the TaskLog.Reader as is done today in for getting tasklogs from the
The advantage is that it would be fairly simple to implement, wouldn't need to make any change to TaskLog.Reader and also would fit along some accepted idioms (e.g. hadoop job -events) ...
What do you think? Would that satisfy your needs?
Others?
> userlogs reader
> ---------------
>
> Key: HADOOP-1181
> URL: https://issues.apache.org/jira/browse/HADOOP-1181
> Project: Hadoop
> Issue Type: Improvement
> Reporter: stack@archive.org
> Attachments: hadoop1181-v2.patch, hadoop1181.patch
>
>
> My jobs output lots of logging. I want to be able to quickly parse the logs across the cluster for anomalies. org.apache.hadoop.tool.Logalyzer looks promising at first but it does not know how to deal with the userlog format and it wants to first copy all logs local. Digging, there does not seem to currently be a reader for hadoop userlog format. TaskLog$Reader is not generally accessible and it too expects logs to be on the local filesystem (The latter is of little good if I want to run the analysis as a mapreduce job).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-1181) userlogs reader
Posted by "stack@archive.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack@archive.org resolved HADOOP-1181.
---------------------------------------
Resolution: Won't Fix
The focus on reading userlogs has been moved to hadoop-1199.
> userlogs reader
> ---------------
>
> Key: HADOOP-1181
> URL: https://issues.apache.org/jira/browse/HADOOP-1181
> Project: Hadoop
> Issue Type: Improvement
> Reporter: stack@archive.org
> Attachments: hadoop1181-v2.patch, hadoop1181-v3.patch, hadoop1181.patch
>
>
> My jobs output lots of logging. I want to be able to quickly parse the logs across the cluster for anomalies. org.apache.hadoop.tool.Logalyzer looks promising at first but it does not know how to deal with the userlog format and it wants to first copy all logs local. Digging, there does not seem to currently be a reader for hadoop userlog format. TaskLog$Reader is not generally accessible and it too expects logs to be on the local filesystem (The latter is of little good if I want to run the analysis as a mapreduce job).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1181) userlogs reader
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486445 ]
Doug Cutting commented on HADOOP-1181:
--------------------------------------
> Actually (we probably should take this discussion elsewhere, so +1 for a separate issue)
Done. This is now HADOOP-1199.
> the idea of starting another mapred job to get the logs of a previous mapred job doesn't seem right ...
It's not to get the logs, but to process the logs and get summary statistics, like the most frequent warnings logged. We shouldn't force folks to copy logs to DFS to determine that.
> userlogs reader
> ---------------
>
> Key: HADOOP-1181
> URL: https://issues.apache.org/jira/browse/HADOOP-1181
> Project: Hadoop
> Issue Type: Improvement
> Reporter: stack@archive.org
> Attachments: hadoop1181-v2.patch, hadoop1181.patch
>
>
> My jobs output lots of logging. I want to be able to quickly parse the logs across the cluster for anomalies. org.apache.hadoop.tool.Logalyzer looks promising at first but it does not know how to deal with the userlog format and it wants to first copy all logs local. Digging, there does not seem to currently be a reader for hadoop userlog format. TaskLog$Reader is not generally accessible and it too expects logs to be on the local filesystem (The latter is of little good if I want to run the analysis as a mapreduce job).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1181) userlogs reader
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486448 ]
Doug Cutting commented on HADOOP-1181:
--------------------------------------
> Get all tasklogs for a given jobid
> $ hadoop job <id> -tasklogs
The natural use of this is for debugging, right? So one might pipe this into 'grep | sort | uniq -c'. Except, for a big job, that's not scalable. So personally I'd prioritize HADOOP-1199 ahead of this. Or am I misunderstanding how this will be used?
> userlogs reader
> ---------------
>
> Key: HADOOP-1181
> URL: https://issues.apache.org/jira/browse/HADOOP-1181
> Project: Hadoop
> Issue Type: Improvement
> Reporter: stack@archive.org
> Attachments: hadoop1181-v2.patch, hadoop1181.patch
>
>
> My jobs output lots of logging. I want to be able to quickly parse the logs across the cluster for anomalies. org.apache.hadoop.tool.Logalyzer looks promising at first but it does not know how to deal with the userlog format and it wants to first copy all logs local. Digging, there does not seem to currently be a reader for hadoop userlog format. TaskLog$Reader is not generally accessible and it too expects logs to be on the local filesystem (The latter is of little good if I want to run the analysis as a mapreduce job).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1181) userlogs reader
Posted by "stack@archive.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack@archive.org updated HADOOP-1181:
--------------------------------------
Attachment: hadoop1181-v2.patch
This version is better than previous attachment.
> userlogs reader
> ---------------
>
> Key: HADOOP-1181
> URL: https://issues.apache.org/jira/browse/HADOOP-1181
> Project: Hadoop
> Issue Type: Improvement
> Reporter: stack@archive.org
> Attachments: hadoop1181-v2.patch, hadoop1181.patch
>
>
> My jobs output lots of logging. I want to be able to quickly parse the logs across the cluster for anomalies. org.apache.hadoop.tool.Logalyzer looks promising at first but it does not know how to deal with the userlog format and it wants to first copy all logs local. Digging, there does not seem to currently be a reader for hadoop userlog format. TaskLog$Reader is not generally accessible and it too expects logs to be on the local filesystem (The latter is of little good if I want to run the analysis as a mapreduce job).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1181) userlogs reader
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486452 ]
Arun C Murthy commented on HADOOP-1181:
---------------------------------------
I misunderstood the feature you were proposing Doug; I agree HADOOP-1199 is far more useful... my bad.
My idea was to only fetch the (relatively small) logs and examine them on the console.
> userlogs reader
> ---------------
>
> Key: HADOOP-1181
> URL: https://issues.apache.org/jira/browse/HADOOP-1181
> Project: Hadoop
> Issue Type: Improvement
> Reporter: stack@archive.org
> Attachments: hadoop1181-v2.patch, hadoop1181.patch
>
>
> My jobs output lots of logging. I want to be able to quickly parse the logs across the cluster for anomalies. org.apache.hadoop.tool.Logalyzer looks promising at first but it does not know how to deal with the userlog format and it wants to first copy all logs local. Digging, there does not seem to currently be a reader for hadoop userlog format. TaskLog$Reader is not generally accessible and it too expects logs to be on the local filesystem (The latter is of little good if I want to run the analysis as a mapreduce job).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1181) userlogs reader
Posted by "stack@archive.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack@archive.org updated HADOOP-1181:
--------------------------------------
Attachment: hadoop1181.patch
Attached is a patch that changes TaskLog$Reader so it uses URLs instead of the file system. It also:
+ Adds a constructor that takes a userlog subdirectory URL.
+ Adds a public getInputStream method that streams over all userlog parts.
+ Makes TaskLog and TaskLog$Reader public rather than default access
+ Adds a main that takes a URL and that then prints to stdout the concatenated logs
I'll not mark this issue as 'patch ready' until others have had a gander. Would be great if Arun C Murthy could review since he wrote the original. In particular, it would be nice to know if the calculation of totalLogSize in the TaskLog$Reader#fetchAll method -- around line 384 in r523437 -- is important. If not, then some near-duplicate code could be replaced with call to the new getInputStream in a version2 of this patch.
> userlogs reader
> ---------------
>
> Key: HADOOP-1181
> URL: https://issues.apache.org/jira/browse/HADOOP-1181
> Project: Hadoop
> Issue Type: Improvement
> Reporter: stack@archive.org
> Attachments: hadoop1181.patch
>
>
> My jobs output lots of logging. I want to be able to quickly parse the logs across the cluster for anomalies. org.apache.hadoop.tool.Logalyzer looks promising at first but it does not know how to deal with the userlog format and it wants to first copy all logs local. Digging, there does not seem to currently be a reader for hadoop userlog format. TaskLog$Reader is not generally accessible and it too expects logs to be on the local filesystem (The latter is of little good if I want to run the analysis as a mapreduce job).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1181) userlogs reader
Posted by "stack@archive.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack@archive.org updated HADOOP-1181:
--------------------------------------
Attachment: hadoop1181-v3.patch
I like both of the above suggestions. Arun's integrates the pulling of remote userlogs nicely into the command-line and if Dougs' HADOOP-1199 had been in place, I wouldn't have filed this issue in the first place.
I would suggest though that the attached patch may still have some (minor) merit: it makes it possible to get a stream on a remote hosts' userlogs dir for grepping, sorting, etc., w/o having to first copy all logs local (or to HDFS) or go via a head node (as per Arun's suggestion) or w/o having to write a mapreduce job as per the Doug suggestion.
Thanks lads
(Redoing patch because v2 had some pollution).
> userlogs reader
> ---------------
>
> Key: HADOOP-1181
> URL: https://issues.apache.org/jira/browse/HADOOP-1181
> Project: Hadoop
> Issue Type: Improvement
> Reporter: stack@archive.org
> Attachments: hadoop1181-v2.patch, hadoop1181-v3.patch, hadoop1181.patch
>
>
> My jobs output lots of logging. I want to be able to quickly parse the logs across the cluster for anomalies. org.apache.hadoop.tool.Logalyzer looks promising at first but it does not know how to deal with the userlog format and it wants to first copy all logs local. Digging, there does not seem to currently be a reader for hadoop userlog format. TaskLog$Reader is not generally accessible and it too expects logs to be on the local filesystem (The latter is of little good if I want to run the analysis as a mapreduce job).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.