You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by 남윤민 <ro...@dgist.ac.kr> on 2013/10/07 05:56:40 UTC
About debugging HDFS and MapReduce...
Hello!!
My name is Yoonmin Nam from South Korea.Currently, I am trying to debug namenode and datanode using eclipse and fully-distributed environemnt in my lab. (1 master and 10 nodes)However, I can do debug but not fully, just some portion of source code of hadoop.This means I cannot see how some daemon thread such as DataStreamer.Also, I want to see how NameNode thread and DataNode thread work but I can't.As you can see, there is Daemon thread (Thread-2) and it might be DataStreamer thread but.. I can't see. So, I think everyone in the hdfs-dev have very professional experience in developing HDFS and MapReduce and I want to ask you about how can I see all real daemons are working for academic reason.Even you have no idea about it, may I ask you that how you test your implementation of HDFS or MapReduce?Thanks!!
// Yoonmin Nam
Re: About debugging HDFS and MapReduce...
Posted by Bobby Evans <ev...@yahoo-inc.com>.
I am not sure what you mean by some portion of source code of Hadoop.
Because you are running on a fully distributed environment you need to
make sure that eclipse is attached to the correct process. If you are
trying to debug the data upload pipeline then you need to be sure that
eclipse is attached to the proper data node. It is hard to guess which
one because you setup has 10 to choose from. Your image showed you
attached to the client, you have to use remote debugging to debug a
different process.
The next thing you are likely to run into is timeout issues. Almost all
of the code has a lot of timeouts, to be able to detect when another
process has stopped responding. This is a bigger problem in MR where you
will be walking through code and suddenly the process disappears because
it was shot be the Task Tracker or Node Manager. I am not as familiar
with the HDFS timeouts, but you probably want to be aware that they exist.
--Bobby
On 10/7/13 12:38 AM, "Yoonmin Nam" <ro...@dgist.ac.kr> wrote:
>Sorry for this inconvenience.
>I uploaded image file on our lab server.
>
>http://infolab.dgist.ac.kr/~ronymin/pictures/debug.png
>
>-----Original Message-----
>From: haosdent [mailto:haosdent@gmail.com]
>Sent: Monday, October 07, 2013 2:24 PM
>To: hdfs-dev@hadoop.apache.org; 남윤민
>Subject: Re: About debugging HDFS and MapReduce...
>
>I still couldn't see the image. The page shows that "The page you
>requested could not be found.".
>
>--
>Best Regards,
>Haosong Huang
>Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
>On Monday, October 7, 2013 at 12:38 PM, 남윤민 wrote:
>
>> Here is image I uploaded.
>>
>> Sorry for that!
>>
>> http://mail.dgist.ac.kr/design/common/image/attaches//ronymin_167_inli
>> ne0_6806441608437847.png
>> > --- Original Message ---
>> > From : "Ted Yu"<yuzhihong@gmail.com (mailto:yuzhihong@gmail.com)> To
>> > : hdfs-dev@hadoop.apache.org (mailto:hdfs-dev@hadoop.apache.org)
>> > Cc : hdfs-dev@hadoop.apache.org (mailto:hdfs-dev@hadoop.apache.org)
>> > Date : 2013/10/07 월요일 오후 1:22:45
>> > Subject : Re: About debugging HDFS and MapReduce...
>> >
>> > Your image didn't go through.
>> > You may want to upload it to image sharing site.
>> >
>> > What version of Hadoop are you using ?
>> >
>> > Thanks
>> >
>> > On Oct 6, 2013, at 8:56 PM, 남윤민 <ronymin@dgist.ac.kr
>>(mailto:ronymin@dgist.ac.kr)> wrote:
>> > > Hello!!
>> > > My name is Yoonmin Nam from South Korea.
>> > >
>> > > Currently, I am trying to debug namenode and datanode using
>> > > eclipse and fully-distributed environemnt in my lab. (1 master and
>> > > 10 nodes)
>> > >
>> > > However, I can do debug but not fully, just some portion of source
>>code of hadoop.
>> > >
>> > > This means I cannot see how some daemon thread such as DataStreamer.
>> > > Also, I want to see how NameNode thread and DataNode thread work
>>but I can't.
>> > >
>> > > As you can see, there is Daemon thread (Thre ad-2) and it might be
>>DataStreamer thread but.. I can't see.
>> > >
>> > >
>> > >
>> > >
>> > > So, I think everyone in the hdfs-dev have very professional
>>experience in developing HDFS and MapReduce and I want to ask you about
>>how can I see all real daemons are working for academic reason.
>> > >
>> > > Even you have no idea about it, may I ask you that how you test
>>your implementation of HDFS or MapReduce?
>> > >
>> > >
>> > > Thanks!!
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > // Yoonmin Nam
>> > >
>>
>>
>> // Yoonmin Nam
>>
>>
>
>
>
>
RE: About debugging HDFS and MapReduce...
Posted by Yoonmin Nam <ro...@dgist.ac.kr>.
Sorry for this inconvenience.
I uploaded image file on our lab server.
http://infolab.dgist.ac.kr/~ronymin/pictures/debug.png
-----Original Message-----
From: haosdent [mailto:haosdent@gmail.com]
Sent: Monday, October 07, 2013 2:24 PM
To: hdfs-dev@hadoop.apache.org; 남윤민
Subject: Re: About debugging HDFS and MapReduce...
I still couldn't see the image. The page shows that "The page you requested could not be found.".
--
Best Regards,
Haosong Huang
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Monday, October 7, 2013 at 12:38 PM, 남윤민 wrote:
> Here is image I uploaded.
>
> Sorry for that!
>
> http://mail.dgist.ac.kr/design/common/image/attaches//ronymin_167_inli
> ne0_6806441608437847.png
> > --- Original Message ---
> > From : "Ted Yu"<yuzhihong@gmail.com (mailto:yuzhihong@gmail.com)> To
> > : hdfs-dev@hadoop.apache.org (mailto:hdfs-dev@hadoop.apache.org)
> > Cc : hdfs-dev@hadoop.apache.org (mailto:hdfs-dev@hadoop.apache.org)
> > Date : 2013/10/07 월요일 오후 1:22:45
> > Subject : Re: About debugging HDFS and MapReduce...
> >
> > Your image didn't go through.
> > You may want to upload it to image sharing site.
> >
> > What version of Hadoop are you using ?
> >
> > Thanks
> >
> > On Oct 6, 2013, at 8:56 PM, 남윤민 <ronymin@dgist.ac.kr (mailto:ronymin@dgist.ac.kr)> wrote:
> > > Hello!!
> > > My name is Yoonmin Nam from South Korea.
> > >
> > > Currently, I am trying to debug namenode and datanode using
> > > eclipse and fully-distributed environemnt in my lab. (1 master and
> > > 10 nodes)
> > >
> > > However, I can do debug but not fully, just some portion of source code of hadoop.
> > >
> > > This means I cannot see how some daemon thread such as DataStreamer.
> > > Also, I want to see how NameNode thread and DataNode thread work but I can't.
> > >
> > > As you can see, there is Daemon thread (Thre ad-2) and it might be DataStreamer thread but.. I can't see.
> > >
> > >
> > >
> > >
> > > So, I think everyone in the hdfs-dev have very professional experience in developing HDFS and MapReduce and I want to ask you about how can I see all real daemons are working for academic reason.
> > >
> > > Even you have no idea about it, may I ask you that how you test your implementation of HDFS or MapReduce?
> > >
> > >
> > > Thanks!!
> > >
> > >
> > >
> > >
> > >
> > > // Yoonmin Nam
> > >
>
>
> // Yoonmin Nam
>
>
Re: About debugging HDFS and MapReduce...
Posted by haosdent <ha...@gmail.com>.
I still couldn't see the image. The page shows that "The page you requested could not be found.".
--
Best Regards,
Haosong Huang
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Monday, October 7, 2013 at 12:38 PM, 남윤민 wrote:
> Here is image I uploaded.
>
> Sorry for that!
>
> http://mail.dgist.ac.kr/design/common/image/attaches//ronymin_167_inline0_6806441608437847.png
> > --- Original Message ---
> > From : "Ted Yu"<yuzhihong@gmail.com (mailto:yuzhihong@gmail.com)>
> > To : hdfs-dev@hadoop.apache.org (mailto:hdfs-dev@hadoop.apache.org)
> > Cc : hdfs-dev@hadoop.apache.org (mailto:hdfs-dev@hadoop.apache.org)
> > Date : 2013/10/07 월요일 오후 1:22:45
> > Subject : Re: About debugging HDFS and MapReduce...
> >
> > Your image didn't go through.
> > You may want to upload it to image sharing site.
> >
> > What version of Hadoop are you using ?
> >
> > Thanks
> >
> > On Oct 6, 2013, at 8:56 PM, 남윤민 <ronymin@dgist.ac.kr (mailto:ronymin@dgist.ac.kr)> wrote:
> > > Hello!!
> > > My name is Yoonmin Nam from South Korea.
> > >
> > > Currently, I am trying to debug namenode and datanode using eclipse and fully-distributed environemnt in my lab. (1 master and 10 nodes)
> > >
> > > However, I can do debug but not fully, just some portion of source code of hadoop.
> > >
> > > This means I cannot see how some daemon thread such as DataStreamer.
> > > Also, I want to see how NameNode thread and DataNode thread work but I can't.
> > >
> > > As you can see, there is Daemon thread (Thre ad-2) and it might be DataStreamer thread but.. I can't see.
> > >
> > >
> > >
> > >
> > > So, I think everyone in the hdfs-dev have very professional experience in developing HDFS and MapReduce and I want to ask you about how can I see all real daemons are working for academic reason.
> > >
> > > Even you have no idea about it, may I ask you that how you test your implementation of HDFS or MapReduce?
> > >
> > >
> > > Thanks!!
> > >
> > >
> > >
> > >
> > >
> > > // Yoonmin Nam
> > >
>
>
> // Yoonmin Nam
>
>
Re: Re: About debugging HDFS and MapReduce...
Posted by 남윤민 <ro...@dgist.ac.kr>.
Here is image I uploaded.
Sorry for that!
http://mail.dgist.ac.kr/design/common/image/attaches//ronymin_167_inline0_6806441608437847.png
--- Original Message ---
>From : "Ted Yu"<yu...@gmail.com>
To : hdfs-dev@hadoop.apache.org
Cc : hdfs-dev@hadoop.apache.org
Date : 2013/10/07 월요일 오후 1:22:45
Subject : Re: About debugging HDFS and MapReduce...
Your image didn't go through.
You may want to upload it to image sharing site.
What version of Hadoop are you using ?
Thanks
On Oct 6, 2013, at 8:56 PM, 남윤민 <ro...@dgist.ac.kr>
// Yoonmin Nam
Re: About debugging HDFS and MapReduce...
Posted by Ted Yu <yu...@gmail.com>.
Your image didn't go through.
You may want to upload it to image sharing site.
What version of Hadoop are you using ?
Thanks
On Oct 6, 2013, at 8:56 PM, 남윤민 <ro...@dgist.ac.kr> wrote:
> Hello!!
> My name is Yoonmin Nam from South Korea.
>
> Currently, I am trying to debug namenode and datanode using eclipse and fully-distributed environemnt in my lab. (1 master and 10 nodes)
>
> However, I can do debug but not fully, just some portion of source code of hadoop.
>
> This means I cannot see how some daemon thread such as DataStreamer.
> Also, I want to see how NameNode thread and DataNode thread work but I can't.
>
> As you can see, there is Daemon thread (Thre ad-2) and it might be DataStreamer thread but.. I can't see.
>
>
>
>
> So, I think everyone in the hdfs-dev have very professional experience in developing HDFS and MapReduce and I want to ask you about how can I see all real daemons are working for academic reason.
>
> Even you have no idea about it, may I ask you that how you test your implementation of HDFS or MapReduce?
>
>
> Thanks!!
>
>
>
>
>
> // Yoonmin Nam
>