You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Xiaowei Li <se...@gmail.com> on 2013/01/11 19:28:45 UTC
log server for hadoop MR jobs??
Hi,
is there a facility, or a log server, to collect all log generated from
each mapper/reducer of a MR job, in a centralized place, or unified log
file on HDFS?
now i look at logs in a naive way: go to jobtracker to check each task....
thanks
-xw
Re: log server for hadoop MR jobs??
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
To add to that, log aggregation is a feature available with Hadoop 2.0
(where mapreduce is re-written to YARN). The functionality is available via
the History Server:
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/HistoryServerRest.html
Thanks
hemanth
On Sat, Jan 12, 2013 at 12:08 AM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
> Have a look on flume..
>
>
> On Fri, Jan 11, 2013 at 11:58 PM, Xiaowei Li <se...@gmail.com> wrote:
>
>> ct all log generated from
>
>
>
>
>
>
> ∞
> Shashwat Shriparv
>
>
Re: log server for hadoop MR jobs??
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
To add to that, log aggregation is a feature available with Hadoop 2.0
(where mapreduce is re-written to YARN). The functionality is available via
the History Server:
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/HistoryServerRest.html
Thanks
hemanth
On Sat, Jan 12, 2013 at 12:08 AM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
> Have a look on flume..
>
>
> On Fri, Jan 11, 2013 at 11:58 PM, Xiaowei Li <se...@gmail.com> wrote:
>
>> ct all log generated from
>
>
>
>
>
>
> ∞
> Shashwat Shriparv
>
>
Re: log server for hadoop MR jobs??
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
To add to that, log aggregation is a feature available with Hadoop 2.0
(where mapreduce is re-written to YARN). The functionality is available via
the History Server:
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/HistoryServerRest.html
Thanks
hemanth
On Sat, Jan 12, 2013 at 12:08 AM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
> Have a look on flume..
>
>
> On Fri, Jan 11, 2013 at 11:58 PM, Xiaowei Li <se...@gmail.com> wrote:
>
>> ct all log generated from
>
>
>
>
>
>
> ∞
> Shashwat Shriparv
>
>
Re: log server for hadoop MR jobs??
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
To add to that, log aggregation is a feature available with Hadoop 2.0
(where mapreduce is re-written to YARN). The functionality is available via
the History Server:
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/HistoryServerRest.html
Thanks
hemanth
On Sat, Jan 12, 2013 at 12:08 AM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
> Have a look on flume..
>
>
> On Fri, Jan 11, 2013 at 11:58 PM, Xiaowei Li <se...@gmail.com> wrote:
>
>> ct all log generated from
>
>
>
>
>
>
> ∞
> Shashwat Shriparv
>
>
Re: log server for hadoop MR jobs??
Posted by shashwat shriparv <dw...@gmail.com>.
Have a look on flume..
On Fri, Jan 11, 2013 at 11:58 PM, Xiaowei Li <se...@gmail.com> wrote:
> ct all log generated from
∞
Shashwat Shriparv
Re: log server for hadoop MR jobs??
Posted by shashwat shriparv <dw...@gmail.com>.
Have a look on flume..
On Fri, Jan 11, 2013 at 11:58 PM, Xiaowei Li <se...@gmail.com> wrote:
> ct all log generated from
∞
Shashwat Shriparv
Re: log server for hadoop MR jobs??
Posted by shashwat shriparv <dw...@gmail.com>.
Have a look on flume..
On Fri, Jan 11, 2013 at 11:58 PM, Xiaowei Li <se...@gmail.com> wrote:
> ct all log generated from
∞
Shashwat Shriparv
Re: log server for hadoop MR jobs??
Posted by shashwat shriparv <dw...@gmail.com>.
Have a look on flume..
On Fri, Jan 11, 2013 at 11:58 PM, Xiaowei Li <se...@gmail.com> wrote:
> ct all log generated from
∞
Shashwat Shriparv