debug mapreduce basics

[Mike Spreitzer <ms...@us.ibm.com>]

So I wrote my first org.apache.hadoop.madpreduce.Job (not ...mapred.Job). 
Oddly enough, when the reducer is invoked, the "Iterable values" parameter 
actually iterates over just one value, not all the corresponding values 
that were output by my map tasks.  I know this from adding calls to 
System.out.println to my map and reduce code.  I also know that my 
combiner was not invoked (also from println debugging).  So now I am 
trying to determine why things are not being wired up correctly.  What is 
the right way to debug situations like this?

Thanks,
Mike Spreitzer
SMTP: mspreitz@us.ibm.com, Lotus Notes: Mike Spreitzer/Watson/IBM
Office phone: +1-914-784-6424 (IBM T/L 863-)
AOL Instant Messaging: M1k3Sprtzr

Re: debug mapreduce basics

[Jeremy Lewi <je...@lewi.us>]

Mike,

Check out this wiki
http://code.google.com/p/hadoop-clusternet/wiki/DebuggingJobsUsingEclipse

It shows how if your running in stand alone mode you can run a job in
debug mode so that you can then start a remote debugging session with
Eclipse. You can then step through your code.

I've found it to be quite helpful.

J
On Thu, 2011-04-28 at 14:59 -0400, Mike Spreitzer wrote:
> So I wrote my first org.apache.hadoop.madpreduce.Job
> (not ...mapred.Job).  Oddly enough, when the reducer is invoked, the
> "Iterable values" parameter actually iterates over just one value, not
> all the corresponding values that were output by my map tasks.  I know
> this from adding calls to System.out.println to my map and reduce
> code.  I also know that my combiner was not invoked (also from println
> debugging).  So now I am trying to determine why things are not being
> wired up correctly.  What is the right way to debug situations like
> this? 
> 
> Thanks, 
> Mike Spreitzer
> SMTP: mspreitz@us.ibm.com, Lotus Notes: Mike Spreitzer/Watson/IBM
> Office phone: +1-914-784-6424 (IBM T/L 863-)
> AOL Instant Messaging: M1k3Sprtzr