You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Sh...@cognizant.com on 2011/09/15 06:09:23 UTC
Debugging Mapreduce programs
Hi
Can we use sysouts in java to debug MR programs.
Where would those sysouts be printed?
What would be the best way to debug a remote system where I have access
to the machine only through putty.
Regards
Shreya
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
prohibited and may be unlawful.
Re: Debugging Mapreduce programs
Posted by Joey Echeverria <jo...@cloudera.com>.
You can also use mrunit [1] to write unit tests against your MapReduce code.
-Joey
[1] http://incubator.apache.org/mrunit/
On Thu, Sep 15, 2011 at 1:18 AM, Subroto Sanyal <su...@huawei.com>wrote:
> **
>
> Hi,****
>
> ** **
>
> MapReduce framework provide different in built approaches to debug a Job:*
> ***
>
> **1) **LocalJobRunner (Running the Job without a cluster; check
> basic functionality of the Job implementation)****
>
> **2) **IsolationRunner (Running a failed TaskAttempt on cluster-
> Remote debugging can be done)****
>
> ** **
>
> Apart from this we can use Sysouts and loggings in the Map or Reduce Tasks.
> ****
>
> These logs will come in “sysout”,”syslog”,”syserr” accordingly. These files
> will be present in the userlogs folder.****
>
> ** **
>
> User can pass Remote Debug argument to Child Process to enable remote
> debugging of Child process.****
>
> If you have access through putty, then you can use remote Debug option and
> connect through eclipse as well.****
>
> ** **
>
> ** **
>
> *Regards,*
> Subroto Sanyal****
> ------------------------------
>
> *From:* bejoy.hadoop@gmail.com [mailto:bejoy.hadoop@gmail.com]
> *Sent:* Thursday, September 15, 2011 9:49 AM
>
> *To:* **mapreduce-user@hadoop.apache.org**
> *Subject:* Re: Debugging Mapreduce programs****
>
> ** **
>
> Shreya
> There are multiple ways to debug map reduce programs. I found comfortable
> using the jobtracker web UI. You can your job tracker web UI at
> http://<server ip>:50030/jobtracker.jsp
> Open it in a browser.
> There you can find the running,failed and retired jobs. You can drill down
> on any job to task tracker log levels from there.
> Mostly I get the root cause of my job failures from there.
>
> Hope it helps****
>
> Regards
> Bejoy K S****
> ------------------------------
>
> *From: *<Sh...@cognizant.com> ****
>
> *Date: *Thu, 15 Sep 2011 09:39:23 +0530****
>
> *To: *<**mapreduce-user@hadoop.apache.org**>****
>
> *ReplyTo: ***mapreduce-user@hadoop.apache.org** ****
>
> *Subject: *Debugging Mapreduce programs****
>
> ** **
>
> Hi****
>
> ** **
>
> Can we use sysouts in java to debug MR programs.****
>
> Where would those sysouts be printed?****
>
> ** **
>
> What would be the best way to debug a remote system where I have access to
> the machine only through putty.****
>
> ** **
>
> Regards****
>
> Shreya****
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information.
> If you are not the intended recipient, please contact the sender by reply
> e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding,
> printing or copying of this email or any action taken in reliance on this
> e-mail is strictly
> prohibited and may be unlawful.
>
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434
RE: Debugging Mapreduce programs
Posted by Subroto Sanyal <su...@huawei.com>.
Hi,
MapReduce framework provide different in built approaches to debug a Job:
1) LocalJobRunner (Running the Job without a cluster; check basic
functionality of the Job implementation)
2) IsolationRunner (Running a failed TaskAttempt on cluster- Remote
debugging can be done)
Apart from this we can use Sysouts and loggings in the Map or Reduce Tasks.
These logs will come in "sysout","syslog","syserr" accordingly. These files
will be present in the userlogs folder.
User can pass Remote Debug argument to Child Process to enable remote
debugging of Child process.
If you have access through putty, then you can use remote Debug option and
connect through eclipse as well.
Regards,
Subroto Sanyal
_____
From: bejoy.hadoop@gmail.com [mailto:bejoy.hadoop@gmail.com]
Sent: Thursday, September 15, 2011 9:49 AM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Debugging Mapreduce programs
Shreya
There are multiple ways to debug map reduce programs. I found comfortable
using the jobtracker web UI. You can your job tracker web UI at
http://<server ip>:50030/jobtracker.jsp
Open it in a browser.
There you can find the running,failed and retired jobs. You can drill down
on any job to task tracker log levels from there.
Mostly I get the root cause of my job failures from there.
Hope it helps
Regards
Bejoy K S
_____
From: <Sh...@cognizant.com>
Date: Thu, 15 Sep 2011 09:39:23 +0530
To: <ma...@hadoop.apache.org>
ReplyTo: mapreduce-user@hadoop.apache.org
Subject: Debugging Mapreduce programs
Hi
Can we use sysouts in java to debug MR programs.
Where would those sysouts be printed?
What would be the best way to debug a remote system where I have access to
the machine only through putty.
Regards
Shreya
This e-mail and any files transmitted with it are for the sole use of the
intended recipient(s) and may contain confidential and privileged
information.
If you are not the intended recipient, please contact the sender by reply
e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding,
printing or copying of this email or any action taken in reliance on this
e-mail is strictly
prohibited and may be unlawful.
Re: Debugging Mapreduce programs
Posted by be...@gmail.com.
Shreya
There are multiple ways to debug map reduce programs. I found comfortable using the jobtracker web UI. You can your job tracker web UI at
http://<server ip>:50030/jobtracker.jsp
Open it in a browser.
There you can find the running,failed and retired jobs. You can drill down on any job to task tracker log levels from there.
Mostly I get the root cause of my job failures from there.
Hope it helps
Regards
Bejoy K S
-----Original Message-----
From: <Sh...@cognizant.com>
Date: Thu, 15 Sep 2011 09:39:23
To: <ma...@hadoop.apache.org>
Reply-To: mapreduce-user@hadoop.apache.org
Subject: Debugging Mapreduce programs
Hi
Can we use sysouts in java to debug MR programs.
Where would those sysouts be printed?
What would be the best way to debug a remote system where I have access
to the machine only through putty.
Regards
Shreya
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
prohibited and may be unlawful.