Hadoop debugging

[Thomas FRIOL <th...@anyware-tech.com>]

Hi all,

I am a new hadoop user and I am now writting my own map reduce 
operations but it is hard for me to find out where comes from the 
problem when the job fails.
So my question is : What is the best way to debug a map reduce job ?

Actually I try to log debug messages using LOG4J but not all of my 
messages are logged and I do not know why.
I also try to throw exceptions when something goes wrong but I am not 
really sure all exceptions are very well catched by the hadoop system. 
Am I wrong ?

I will appreciate some tips and advices to be able to increase my 
productivity in developping map and reduce jobs.

Thanks for any help.

Thomas.

-- 
Thomas FRIOL
Développeur Eclipse / Eclipse Developer
Solutions & Technologies
ANYWARE TECHNOLOGIES
Tél      : +33 (0)561 000 653
Portable : +33 (0)609 704 810
Fax      : +33 (0)561 005 146
www.anyware-tech.com

Re: Hadoop debugging

[Owen O'Malley <ow...@yahoo-inc.com>]

On Jul 17, 2006, at 1:36 AM, Thomas FRIOL wrote:

> Hi all,
>
> I am a new hadoop user and I am now writting my own map reduce 
> operations but it is hard for me to find out where comes from the 
> problem when the job fails.
> So my question is : What is the best way to debug a map reduce job ?

Ok, I should probably put this onto a wiki page, but my short answer is:

1. Start by getting everything running (likely on a small input) in the 
local runner. You do this by setting your
job tracker to "local" in your config. The local runner can run under 
the debugger and is not distributed.

2. Run the small input on a 1 node cluster. This will smoke out all of 
the issues that happen with distribution and the "real" task runner, 
but you only have a single place to look at logs. Most useful are the 
task and job tracker logs. Make sure you are logging at the INFO level 
or you will miss clues like the output of your tasks.

3. Run on a big cluster. Recently, I added the keep.failed.task.files 
config variable that tells the system to keep files for tasks that 
fail. This leaves "dead" files around that you can debug with. On the 
node with the failed task, go to the task tracker's local directory and 
cd to <local>/taskTracker/<taskid> and run
% hadoop org.apache.hadoop.IsolationRunner job.xml
This will run the failed task in a single jvm, which can be in the 
debugger, over precisely the same input.

I also have a patch that will let you specify a task to keep, even if 
it doesn't fail. Other than that, logging is your friend.

I don't have issues with my log messages getting through, so you might 
check your filters. Exceptions are mostly handled right, but we've 
found and fixed spots where they weren't, so that is possible. Usually 
it involves someone throwing an unchecked exception like RuntimeError 
and the catch only catching checked exceptions.

-- Owen