Debugging DFS and MapReduce through Eclipse

[Dennis Kubes <nu...@dragonflymc.com>]

Has anyone been able to successfully debug DFS and MapReduce servers 
running though eclipse.  I can get all the servers started and can run 
MapReduce tasks inside of eclipse but I am getting both classpath errors 
and debugging stalls.

I am just curious what kinds of setups people have for doing large scale 
development of DFS and MapReduce?

Dennis

Re: Debugging DFS and MapReduce through Eclipse

[Dennis Kubes <nu...@dragonflymc.com>]

Ben,

Thanks for the response.  I was trying to set it up to run inside of 
eclipse instead of remote debugging.  And I finally got it working.  I 
have both the nutch and hadoop projects pulled from SVN so I am starting 
the dfs and mapreduce servers from the hadoop codebase.  I am starting 
the servers all on one box inside of eclipse.  Setting up the servers to 
run in psuedo-distributed mode through eclipse is pretty easy, you just 
have to have launchers with the correct configurations (parameters). 

If you want to debug the servers and you want to do it inside of eclipse 
and not through remote debugging (such as is the case if you are doing 
it on windows without cygwin), the setup is not so obvious.  The key is 
when you are creating debug launchers, especially for TaskTracker, you 
have to include the nutch project and the nutch project has to export 
its plugin directory and all the jars in the lib directory.  This way 
the TaskTracker can find the correct plugins and source code when 
running tasks.  Again the key is the classpath of the launchers, not of 
the projects themselves.

If anyone needs a more detailed explanation on how to set this up, send 
me an email.

Dennis

Ben Reed wrote:
> I debug using eclipse. For example, if I need to debug JobTracker, I 
> put the following lines in the hadoop script:
>
> elif [ "$COMMAND" = "jobtracker" ] ; then
>   #HADOOP_OPTS="$HADOOP_OPTS 
> -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=y"
>   CLASS=org.apache.hadoop.mapred.JobTracker
>
> the I just start everything up using start-all.sh. I should note that 
> I always debug code built inside of eclipse. I don't use ant. I just 
> import Java projects from SVN and export JAR files.
>
> ben
>
> On May 24, 2006, at 8:49 AM, Dennis Kubes wrote:
>
>> Has anyone been able to successfully debug DFS and MapReduce servers 
>> running though eclipse.  I can get all the servers started and can 
>> run MapReduce tasks inside of eclipse but I am getting both classpath 
>> errors and debugging stalls.
>>
>> I am just curious what kinds of setups people have for doing large 
>> scale development of DFS and MapReduce?
>>
>> Dennis
>>
>>
>

Re: Debugging DFS and MapReduce through Eclipse

[Ben Reed <br...@yahoo-inc.com>]

I debug using eclipse. For example, if I need to debug JobTracker, I  
put the following lines in the hadoop script:

elif [ "$COMMAND" = "jobtracker" ] ; then
   #HADOOP_OPTS="$HADOOP_OPTS - 
agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=y"
   CLASS=org.apache.hadoop.mapred.JobTracker

the I just start everything up using start-all.sh. I should note that  
I always debug code built inside of eclipse. I don't use ant. I just  
import Java projects from SVN and export JAR files.

ben

On May 24, 2006, at 8:49 AM, Dennis Kubes wrote:

> Has anyone been able to successfully debug DFS and MapReduce  
> servers running though eclipse.  I can get all the servers started  
> and can run MapReduce tasks inside of eclipse but I am getting both  
> classpath errors and debugging stalls.
>
> I am just curious what kinds of setups people have for doing large  
> scale development of DFS and MapReduce?
>
> Dennis
>
>