Environment setup for testing a patch submitting jobs from eclipse.

[Thoihen Maibam <th...@gmail.com>]

Hi All,

Apology to everyone in case my question has been dealt before.If the
question has been answered before please do provide me the link .

Basically, I want to start contributing to Hadoop by submitting patches
mostly from Map Reduce issues.

1. Assuming I create a patch, now I want to  test the patch.
2. Basically, I want to work from eclipse and make use of the breakpoints
supported by eclipse.
3. Assuming I ran all the hadoop daemons in single node.
4. Will the eclipse plugin included in hadoop work for submiting the jobs
and hit the breakpoints , this is because I want to trace the code.
5. Normally, how do the Hadoop committers setup their Hadoop development
setup look like. Do they use eclipse to set breakpoints in the eclipse ide.
6. If I use eclipse , learning curve of Hadoop code base would be very easy.

Can somebody guide me how do I submit my job from eclipse and set
breakpoints in Hadoop core code in JobTracker, TaskTracker etc.

Regards
thoihen

Re: Environment setup for testing a patch submitting jobs from eclipse.

[maisnam ns <ma...@gmail.com>]

Hi thoihen,

It is extremely tough to debug the Hadoop core code, but it is not
impossible.In fact debugging through eclipse and setting breakpoints may
help considerably in understanding the flow of hadoop core logic .

Steps are given below:
1. You may have to run all you hadoop daemons in your single machine/laptop
.
2. Lets say you want to debug bin/hadoop namenode -format logic flow.
3. First in your hadoop conf directory find hadoop-env.sh add this line ----
export
HADOOP_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5002"
4. Set your debug configurations and for port add 5002 and localhost as
server
5.Copy the required hadoop source files core/mapred/hdfs etc (assuming
1.0.4 version) in your eclipse project .
6.Search for NameNode.java with ctrl+shift+R and type NameNode.java
7. Open NameNode file , and set a breakpoint in inside the main method ,
say at createNameNode method
8. Now open the terminal cd to hadoop folder type bin/hadoop namenode
-format and run this command when it start as breakpoint is set it will stop
9. Now right click on NameNode.java and Debug As java application and then
click
10. Your breakpoint will be hit at the location where you put your
breakpoint.
11. Press F6 to go line by line or F5 to enter inside the method in this
case createNameNode
12. If you encounter errors  you may have to resolve those issues first.
13. Your breakpoint will be hit and keep on following the logic depending
on your interest to go line by line or inside the method
**


On Sat, Apr 27, 2013 at 1:18 AM, Thoihen Maibam <th...@gmail.com>wrote:

> Hi All,
>
> Apology to everyone in case my question has been dealt before.If the
> question has been answered before please do provide me the link .
>
> Basically, I want to start contributing to Hadoop by submitting patches
> mostly from Map Reduce issues.
>
> 1. Assuming I create a patch, now I want to  test the patch.
> 2. Basically, I want to work from eclipse and make use of the breakpoints
> supported by eclipse.
> 3. Assuming I ran all the hadoop daemons in single node.
> 4. Will the eclipse plugin included in hadoop work for submiting the jobs
> and hit the breakpoints , this is because I want to trace the code.
> 5. Normally, how do the Hadoop committers setup their Hadoop development
> setup look like. Do they use eclipse to set breakpoints in the eclipse ide.
> 6. If I use eclipse , learning curve of Hadoop code base would be very
> easy.
>
> Can somebody guide me how do I submit my job from eclipse and set
> breakpoints in Hadoop core code in JobTracker, TaskTracker etc.
>
> Regards
> thoihen
>