You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Jason Yang <li...@gmail.com> on 2012/07/04 11:25:04 UTC
How to connect to a cluster by using eclipse
Hi, all
I have a hadoop cluster with 3 nodes, the network topology is like this:
1. For each DataNode, its IP address is like :192.168.0.XXX;
2. For the NameNode, it has two network cards: one is connect with the
DataNodes as a local LAN with IP address 192.168.0.110, while the other one
is connect to the company network(which eventually connect to the Internet);
--
now I'm trying to debug a MapReduce program on a computer which is in the
company network. Since the jobtracker in this scenario is 192.168.0.110:9001,
I was wondering how could I connect to the cluster by using eclipse?
--
YANG, Lin
Re: How to connect to a cluster by using eclipse
Posted by Yang <te...@gmail.com>.
in some cases, u have to debug it in a real cluster (for example a
production problem)
in that case you need to specify the
-Xdebug -Xrunjdwp:transport=dt_socket,address=12345,server=n,suspend=n
for mapred.java.child.opts for the job (sorry forgot the exact param name,
maybe not exactly mapred.java.child.opts)
then use eclipse to connect to the tasktracker , note you need to choose
between "Listen/attach" in the
"Run remote application" window in eclipse
On Wed, Jul 4, 2012 at 7:00 PM, Jason Yang <li...@gmail.com> wrote:
> ramon,
>
> Thank for your reply very much.
>
> However, I was still wonder whether I could debug a MR application in this
> way.
>
> I have read some posts talking about using NAT to redirect all the packets
> to the network card which connect to the local LAN, but it does not work as
> I tried to redirect by using iptables :(
>
> 在 2012年7月4日星期三， 写道：
>
>> Jason,
>>
>>
>> the easiest way to debug a MapRedupe program with eclipse is working
>> on hadoop local.
>> http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html#Local In
>> this mode all the components run locally on the same VM and can be easily
>> debugged using Eclipse. Hope this will be useful.
>>
>>
>>
>> *From:* Jason Yang [mailto:lin.yang.jason@gmail.com]
>> *Sent:* miércoles, 04 de julio de 2012 11:25
>> *To:* mapreduce-user
>> *Subject:* How to connect to a cluster by using eclipse
>>
>>
>>
>> Hi, all
>>
>>
>>
>> I have a hadoop cluster with 3 nodes, the network topology is like this:
>>
>> 1. For each DataNode, its IP address is like :192.168.0.XXX;
>>
>> 2. For the NameNode, it has two network cards: one is connect with the
>> DataNodes as a local LAN with IP address 192.168.0.110, while the other one
>> is connect to the company network(which eventually connect to the Internet);
>>
>> --
>>
>> now I'm trying to debug a MapReduce program on a computer which is in the
>> company network. Since the jobtracker in this scenario is
>> 192.168.0.110:9001, I was wondering how could I connect to the cluster
>> by using eclipse?
>>
>>
>>
>> --
>>
>> YANG, Lin
>>
>>
>>
>> ------------------------------
>> Subject to local law, communications with Accenture and its affiliates
>> including telephone calls and emails (including content), may be monitored
>> by our systems for the purposes of security and the assessment of internal
>> compliance with Accenture policy.
>>
>> ______________________________________________________________________________________
>>
>> www.accenture.com
>>
>
>
> --
> YANG, Lin
>
>
Re: How to connect to a cluster by using eclipse
Posted by Jason Yang <li...@gmail.com>.
all right, thanks~
在 2012年7月5日星期四，Marcos Ortiz 写道：
> Jason,
> Ramon is right.
> The best way to debug a MapReduce job is mounting a local cluster, and
> then, when you have tested enough your code, then, you can
> deploy it in a real distributed cluster.
> On 07/04/2012 10:00 PM, Jason Yang wrote:
>
> ramon,
>
> Thank for your reply very much.
>
> However, I was still wonder whether I could debug a MR application in
> this way.
>
> I have read some posts talking about using NAT to redirect all the
> packets to the network card which connect to the local LAN, but it does not
> work as I tried to redirect by using iptables :(
>
> 在 2012年7月4日星期三， 写道：
>
>> Jason,
>>
>>
>> the easiest way to debug a MapRedupe program with eclipse is working on
>> hadoop local.
>> http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html#Local In
>> this mode all the components run locally on the same VM and can be easily
>> debugged using Eclipse. Hope this will be useful.
>>
>>
>>
>> *From:* Jason Yang [mailto:lin.yang.jason@gmail.com]
>> *Sent:* miércoles, 04 de julio de 2012 11:25
>> *To:* mapreduce-user
>> *Subject:* How to connect to a cluster by using eclipse
>>
>>
>>
>> Hi, all
>>
>>
>>
>> I have a hadoop cluster with 3 nodes, the network topology is like this:
>>
>> 1. For each DataNode, its IP address is like :192.168.0.XXX;
>>
>> 2. For the NameNode, it has two network cards: one is connect with the
>> DataNodes as a local LAN with IP address 192.168.0.110, while the other one
>> is connect to the company network(which eventually connect to the Internet);
>>
>> --
>>
>> now I'm trying to debug a MapReduce program on a computer which is in the
>> company network. Since the jobtracker in this scenario is
>> 192.168.0.110:9001, I was wondering how could I connect to the cluster
>> by using eclipse?
>>
>>
>>
>> --
>>
>> YANG, Lin
>>
>>
>>
>> ------------------------------
>> Subject to local law, communications with Accenture and its affiliates
>> including telephone calls and emails (including content), may be monitored
>> by our systems for the purposes of security and the assessment of internal
>> compliance with Accenture policy.
>>
>> ______________________________________________________________________________________
>>
>> www.accenture.com
>>
>
>
> --
> YANG, Lin
>
>
> --
> **
>
> Marcos Luis Ortíz Valmaseda
> *Data Engineer && Sr. System Administrator at UCI*
> about.me/marcosortiz
> My Blog <http://marcosluis2186.posterous.com>
> @marcosluis2186 <http://twitter.com/marcosluis2186>
> **
>
>
>
> <http://www.uci.cu/>
>
>
--
YANG, Lin
Re: How to connect to a cluster by using eclipse
Posted by Marcos Ortiz <ml...@uci.cu>.
Jason,
Ramon is right.
The best way to debug a MapReduce job is mounting a local cluster, and
then, when you have tested enough your code, then, you can
deploy it in a real distributed cluster.
On 07/04/2012 10:00 PM, Jason Yang wrote:
> ramon,
>
> Thank for your reply very much.
>
> However, I was still wonder whether I could debug a MR application in
> this way.
>
> I have read some posts talking about using NAT to redirect all the
> packets to the network card which connect to the local LAN, but it
> does not work as I tried to redirect by using iptables :(
>
> 在 2012年7月4日星期三， 写道：
>
> Jason,
>
>
> the easiest way to debug a MapRedupe program with eclipse is
> working on hadoop local.
> http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html#Local
>
>
> In this mode all the components run locally on the same VM and
> can be easily debugged using Eclipse.
>
>
> Hope this will be useful.
>
> *From:*Jason Yang [mailto:lin.yang.jason@gmail.com
> <javascript:_e({}, 'cvml', 'lin.yang.jason@gmail.com');>]
> *Sent:* miércoles, 04 de julio de 2012 11:25
> *To:* mapreduce-user
> *Subject:* How to connect to a cluster by using eclipse
>
> Hi, all
>
> I have a hadoop cluster with 3 nodes, the network topology is like
> this:
>
> 1. For each DataNode, its IP address is like :192.168.0.XXX;
>
> 2. For the NameNode, it has two network cards: one is connect with
> the DataNodes as a local LAN with IP address 192.168.0.110, while
> the other one is connect to the company network(which eventually
> connect to the Internet);
>
> --
>
> now I'm trying to debug a MapReduce program on a computer which is
> in the company network. Since the jobtracker in this scenario is
> 192.168.0.110:9001 <http://192.168.0.110:9001>, I was wondering
> how could I connect to the cluster by using eclipse?
>
> --
>
> YANG, Lin
>
>
> ------------------------------------------------------------------------
> Subject to local law, communications with Accenture and its
> affiliates including telephone calls and emails (including
> content), may be monitored by our systems for the purposes of
> security and the assessment of internal compliance with Accenture
> policy.
> ______________________________________________________________________________________
>
> www.accenture.com <http://www.accenture.com>
>
>
>
> --
> YANG, Lin
>
--
Marcos Luis Ortíz Valmaseda
*Data Engineer && Sr. System Administrator at UCI*
about.me/marcosortiz <http://about.me/marcosortiz>
My Blog <http://marcosluis2186.posterous.com>
@marcosluis2186 <http://twitter.com/marcosluis2186>
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Re: How to connect to a cluster by using eclipse
Posted by Jason Yang <li...@gmail.com>.
ramon,
Thank for your reply very much.
However, I was still wonder whether I could debug a MR application in this
way.
I have read some posts talking about using NAT to redirect all the packets
to the network card which connect to the local LAN, but it does not work as
I tried to redirect by using iptables :(
在 2012年7月4日星期三， 写道：
> Jason,
>
>
> the easiest way to debug a MapRedupe program with eclipse is working
> on hadoop local.
> http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html#Local In
> this mode all the components run locally on the same VM and can be easily
> debugged using Eclipse. Hope this will be useful.
>
>
>
> *From:* Jason Yang [mailto:lin.yang.jason@gmail.com <javascript:_e({},
> 'cvml', 'lin.yang.jason@gmail.com');>]
> *Sent:* miércoles, 04 de julio de 2012 11:25
> *To:* mapreduce-user
> *Subject:* How to connect to a cluster by using eclipse
>
>
>
> Hi, all
>
>
>
> I have a hadoop cluster with 3 nodes, the network topology is like this:
>
> 1. For each DataNode, its IP address is like :192.168.0.XXX;
>
> 2. For the NameNode, it has two network cards: one is connect with the
> DataNodes as a local LAN with IP address 192.168.0.110, while the other one
> is connect to the company network(which eventually connect to the Internet);
>
> --
>
> now I'm trying to debug a MapReduce program on a computer which is in the
> company network. Since the jobtracker in this scenario is
> 192.168.0.110:9001, I was wondering how could I connect to the cluster by
> using eclipse?
>
>
>
> --
>
> YANG, Lin
>
>
>
> ------------------------------
> Subject to local law, communications with Accenture and its affiliates
> including telephone calls and emails (including content), may be monitored
> by our systems for the purposes of security and the assessment of internal
> compliance with Accenture policy.
>
> ______________________________________________________________________________________
>
> www.accenture.com
>
--
YANG, Lin
RE: How to connect to a cluster by using eclipse
Posted by ra...@accenture.com.
Jason,
the easiest way to debug a MapRedupe program with eclipse is working on hadoop local. http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html#Local
In this mode all the components run locally on the same VM and can be easily debugged using Eclipse.
Hope this will be useful.
From: Jason Yang [mailto:lin.yang.jason@gmail.com]
Sent: miércoles, 04 de julio de 2012 11:25
To: mapreduce-user
Subject: How to connect to a cluster by using eclipse
Hi, all
I have a hadoop cluster with 3 nodes, the network topology is like this:
1. For each DataNode, its IP address is like :192.168.0.XXX;
2. For the NameNode, it has two network cards: one is connect with the DataNodes as a local LAN with IP address 192.168.0.110, while the other one is connect to the company network(which eventually connect to the Internet);
--
now I'm trying to debug a MapReduce program on a computer which is in the company network. Since the jobtracker in this scenario is 192.168.0.110:9001<http://192.168.0.110:9001>, I was wondering how could I connect to the cluster by using eclipse?
--
YANG, Lin
________________________________
Subject to local law, communications with Accenture and its affiliates including telephone calls and emails (including content), may be monitored by our systems for the purposes of security and the assessment of internal compliance with Accenture policy.
______________________________________________________________________________________
www.accenture.com