Re: is there a way to to debug hadoop from Eclipse

[Richard Zhang <ri...@gmail.com>]

>
> Hello Hadoopers:
>
> Is there a way to debug the hadoop code from Eclipse IDE? I am using Eclipse to read the source and build the project now.
> How to start the hadoop jobs from Eclipse? Say if we can put the server names, could we trace the running process through
> eclipse, such as setting breakpoints, check the variable values? That should be very helpful for development.
> If anyone know how to do it, could you please give some info?
> Thanks.
>
> Richard
>
>
>

Re: is there a way to to debug hadoop from Eclipse

["Craig E. Ward" <cr...@computer.org>]

Richard Zhang wrote:
>> Hello Hadoopers:
>>
>> Is there a way to debug the hadoop code from Eclipse IDE? I am using Eclipse to read the source and build the project now.
>> How to start the hadoop jobs from Eclipse? Say if we can put the server names, could we trace the running process through
>> eclipse, such as setting breakpoints, check the variable values? That should be very helpful for development.
>> If anyone know how to do it, could you please give some info?
>> Thanks.
>>
>> Richard

I run my tests in Eclipse without doing anything special (Run->Debug). It's 
only a single node, but it works to debug my code.

There's a plug-in that looks like it could submit jobs to a "real" Hadoop 
cluster, but I couldn't make it work. As it wasn't really necessary, I didn't 
put a lot of effort into it.

I'm using Mac OS X 10.4.11 (only has Java 5) and Eclipse 3.3.2.

-- 
Craig.E.Ward@computer.org

Re: is there a way to to debug hadoop from Eclipse

[Brian Vargas <br...@ardvaark.net>]

JMock also works rather well, using its cglib extensions, for mocking 
out fake FileSystem implementations, if you're expecting your code to 
make calls directly to the filesystem for some reason.

Brian

Matt Kent wrote:
> JMock is a unit testing tool for creating mock objects. I use it to mock
> things like OutputCollector and Reporter, so I can unit test mappers and
> reducers without running a cluster. In other words, I'm just testing the
> logic of the code within the map() and reduce() methods, and testing the
> map and reduce separately. I'm not feeding it real data from HDFS or
> running the code in a real cluster.
> 
> Matt
> 
> On Tue, 2008-06-17 at 18:50 -0700, Richard Zhang wrote:
>> I creates three virtual machines, each of them works as a node.
>> Does the JMock support debugging with multiple nodes cluster within Eclipse?
>> Could we set up breakpoints, trace the running steps of the map reduce
>> program?
>> Richard
>>
>>
>> On Mon, Jun 16, 2008 at 6:54 PM, Matt Kent <ma...@persai.com> wrote:
>>
>>> The approach I've taken is to use JMock and create a unit test for the
>>> mapreduce, then debug that within Eclipse on my workstation. For
>>> performance debugging, I use YourKit on the cluster.
>>>
>>> Matt
>>>
>>> On Mon, 2008-06-16 at 16:58 -0700, Mori Bellamy wrote:
>>>> Hey Richard,
>>>>
>>>> I'm interested in the same thing myself :D. I was researching it
>>>> earlier today, and the best I know to do is to use Eclipse's remote
>>>> debugging functionality (although this won't completely work. each map/
>>>> reduce task spawns on its on JVM, making debugging really hard). but
>>>> if you want, you can debug up until the mappers/reducers spawn. To do
>>>> this, you need to pass certain debug-flags into the JVM. So you'd need
>>>> to do export HADOOP_OPTS=myFLagsForRemoteDebug
>>>> and then you'd go to eclips ->run->open debug dialog and set up remote
>>>> debugging with the correct port.
>>>>
>>>> if you find out a way to debug the mappers/reducers on eclipse, let me
>>>> know :D
>>>>
>>>>
>>>> On Jun 16, 2008, at 3:10 PM, Richard Zhang wrote:
>>>>
>>>>>> Hello Hadoopers:
>>>>>>
>>>>>> Is there a way to debug the hadoop code from Eclipse IDE? I am
>>>>>> using Eclipse to read the source and build the project now.
>>>>>> How to start the hadoop jobs from Eclipse? Say if we can put the
>>>>>> server names, could we trace the running process through
>>>>>> eclipse, such as setting breakpoints, check the variable values?
>>>>>> That should be very helpful for development.
>>>>>> If anyone know how to do it, could you please give some info?
>>>>>> Thanks.
>>>>>>
>>>>>> Richard
>>>>>>
>>>>>>
>>>>>>
>>>
> 

Re: is there a way to to debug hadoop from Eclipse

[Matt Kent <ma...@persai.com>]

JMock is a unit testing tool for creating mock objects. I use it to mock
things like OutputCollector and Reporter, so I can unit test mappers and
reducers without running a cluster. In other words, I'm just testing the
logic of the code within the map() and reduce() methods, and testing the
map and reduce separately. I'm not feeding it real data from HDFS or
running the code in a real cluster.

Matt

On Tue, 2008-06-17 at 18:50 -0700, Richard Zhang wrote:
> I creates three virtual machines, each of them works as a node.
> Does the JMock support debugging with multiple nodes cluster within Eclipse?
> Could we set up breakpoints, trace the running steps of the map reduce
> program?
> Richard
> 
> 
> On Mon, Jun 16, 2008 at 6:54 PM, Matt Kent <ma...@persai.com> wrote:
> 
> > The approach I've taken is to use JMock and create a unit test for the
> > mapreduce, then debug that within Eclipse on my workstation. For
> > performance debugging, I use YourKit on the cluster.
> >
> > Matt
> >
> > On Mon, 2008-06-16 at 16:58 -0700, Mori Bellamy wrote:
> > > Hey Richard,
> > >
> > > I'm interested in the same thing myself :D. I was researching it
> > > earlier today, and the best I know to do is to use Eclipse's remote
> > > debugging functionality (although this won't completely work. each map/
> > > reduce task spawns on its on JVM, making debugging really hard). but
> > > if you want, you can debug up until the mappers/reducers spawn. To do
> > > this, you need to pass certain debug-flags into the JVM. So you'd need
> > > to do export HADOOP_OPTS=myFLagsForRemoteDebug
> > > and then you'd go to eclips ->run->open debug dialog and set up remote
> > > debugging with the correct port.
> > >
> > > if you find out a way to debug the mappers/reducers on eclipse, let me
> > > know :D
> > >
> > >
> > > On Jun 16, 2008, at 3:10 PM, Richard Zhang wrote:
> > >
> > > >>
> > > >> Hello Hadoopers:
> > > >>
> > > >> Is there a way to debug the hadoop code from Eclipse IDE? I am
> > > >> using Eclipse to read the source and build the project now.
> > > >> How to start the hadoop jobs from Eclipse? Say if we can put the
> > > >> server names, could we trace the running process through
> > > >> eclipse, such as setting breakpoints, check the variable values?
> > > >> That should be very helpful for development.
> > > >> If anyone know how to do it, could you please give some info?
> > > >> Thanks.
> > > >>
> > > >> Richard
> > > >>
> > > >>
> > > >>
> > >
> >
> >

Re: is there a way to to debug hadoop from Eclipse

[Richard Zhang <ri...@gmail.com>]

 I creates three virtual machines, each of them works as a node.
Does the JMock support debugging with multiple nodes cluster within Eclipse?
Could we set up breakpoints, trace the running steps of the map reduce
program?
Richard


On Mon, Jun 16, 2008 at 6:54 PM, Matt Kent <ma...@persai.com> wrote:

> The approach I've taken is to use JMock and create a unit test for the
> mapreduce, then debug that within Eclipse on my workstation. For
> performance debugging, I use YourKit on the cluster.
>
> Matt
>
> On Mon, 2008-06-16 at 16:58 -0700, Mori Bellamy wrote:
> > Hey Richard,
> >
> > I'm interested in the same thing myself :D. I was researching it
> > earlier today, and the best I know to do is to use Eclipse's remote
> > debugging functionality (although this won't completely work. each map/
> > reduce task spawns on its on JVM, making debugging really hard). but
> > if you want, you can debug up until the mappers/reducers spawn. To do
> > this, you need to pass certain debug-flags into the JVM. So you'd need
> > to do export HADOOP_OPTS=myFLagsForRemoteDebug
> > and then you'd go to eclips ->run->open debug dialog and set up remote
> > debugging with the correct port.
> >
> > if you find out a way to debug the mappers/reducers on eclipse, let me
> > know :D
> >
> >
> > On Jun 16, 2008, at 3:10 PM, Richard Zhang wrote:
> >
> > >>
> > >> Hello Hadoopers:
> > >>
> > >> Is there a way to debug the hadoop code from Eclipse IDE? I am
> > >> using Eclipse to read the source and build the project now.
> > >> How to start the hadoop jobs from Eclipse? Say if we can put the
> > >> server names, could we trace the running process through
> > >> eclipse, such as setting breakpoints, check the variable values?
> > >> That should be very helpful for development.
> > >> If anyone know how to do it, could you please give some info?
> > >> Thanks.
> > >>
> > >> Richard
> > >>
> > >>
> > >>
> >
>
>

Re: is there a way to to debug hadoop from Eclipse

[Matt Kent <ma...@persai.com>]

The approach I've taken is to use JMock and create a unit test for the
mapreduce, then debug that within Eclipse on my workstation. For
performance debugging, I use YourKit on the cluster.

Matt

On Mon, 2008-06-16 at 16:58 -0700, Mori Bellamy wrote:
> Hey Richard,
> 
> I'm interested in the same thing myself :D. I was researching it  
> earlier today, and the best I know to do is to use Eclipse's remote  
> debugging functionality (although this won't completely work. each map/ 
> reduce task spawns on its on JVM, making debugging really hard). but  
> if you want, you can debug up until the mappers/reducers spawn. To do  
> this, you need to pass certain debug-flags into the JVM. So you'd need  
> to do export HADOOP_OPTS=myFLagsForRemoteDebug
> and then you'd go to eclips ->run->open debug dialog and set up remote  
> debugging with the correct port.
> 
> if you find out a way to debug the mappers/reducers on eclipse, let me  
> know :D
> 
> 
> On Jun 16, 2008, at 3:10 PM, Richard Zhang wrote:
> 
> >>
> >> Hello Hadoopers:
> >>
> >> Is there a way to debug the hadoop code from Eclipse IDE? I am  
> >> using Eclipse to read the source and build the project now.
> >> How to start the hadoop jobs from Eclipse? Say if we can put the  
> >> server names, could we trace the running process through
> >> eclipse, such as setting breakpoints, check the variable values?  
> >> That should be very helpful for development.
> >> If anyone know how to do it, could you please give some info?
> >> Thanks.
> >>
> >> Richard
> >>
> >>
> >>
> 

Re: is there a way to to debug hadoop from Eclipse

[Mori Bellamy <mb...@apple.com>]

Hey Richard,

I'm interested in the same thing myself :D. I was researching it  
earlier today, and the best I know to do is to use Eclipse's remote  
debugging functionality (although this won't completely work. each map/ 
reduce task spawns on its on JVM, making debugging really hard). but  
if you want, you can debug up until the mappers/reducers spawn. To do  
this, you need to pass certain debug-flags into the JVM. So you'd need  
to do export HADOOP_OPTS=myFLagsForRemoteDebug
and then you'd go to eclips ->run->open debug dialog and set up remote  
debugging with the correct port.

if you find out a way to debug the mappers/reducers on eclipse, let me  
know :D


On Jun 16, 2008, at 3:10 PM, Richard Zhang wrote:

>>
>> Hello Hadoopers:
>>
>> Is there a way to debug the hadoop code from Eclipse IDE? I am  
>> using Eclipse to read the source and build the project now.
>> How to start the hadoop jobs from Eclipse? Say if we can put the  
>> server names, could we trace the running process through
>> eclipse, such as setting breakpoints, check the variable values?  
>> That should be very helpful for development.
>> If anyone know how to do it, could you please give some info?
>> Thanks.
>>
>> Richard
>>
>>
>>