You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hama.apache.org by Ashish Agarwal <as...@gmail.com> on 2011/06/11 12:41:22 UTC

Debugging hama in eclipse

Hi,

I have written some code for my GSoC task (Hama-367) and am now trying to
debug.

I am trying to set up the environment into pseudo-distributed mode so I can
debug. I think the value of bsp.master.address in hama-site.xml has to be
local as this means LocalBSPRunner (as localhost would call 127.0.0.1
server), but where do I specify that in the code in eclipse ?

My next question is - what is the difference between MiniBSPCluster and
LocalBSPRunner  and which file should I be using to be able to step through
the code ?

These are the processes currently running on my machine -

6112 NameNode
3319 ZooKeeperRunner
3018
6449 SecondaryNameNode
5709 HMaster
5643 HQuorumPeer
6684 TaskTracker
6526 JobTracker
8074 Jps
6277 DataNode

Thank you,
Ashish

Re: Debugging hama in eclipse

Posted by Thomas Jungblut <th...@googlemail.com>.

Hi,

or an even more sophisticated way to debug:
Actually Eclipse always spawns a new JVM if you're running. In debug mode
eclipse just connects with it to debug at a certain breakpoint.
So you can start the deamons with specific VM arguments and connect with
eclipse.
Here's a really good post of IBM to that theme:
http://www.ibm.com/developerworks/opensource/library/os-eclipse-javadebug/index.html
Or just search with your fav'd engine for: "eclipse remote debugging".

gl :)

2011/6/15 Edward J. Yoon <ed...@apache.org>

> Hi,
>
> I would suggest following steps to debug:
>
> 1. Install the Hadoop 0.20.x on your laptop.
> 2. Run the Hama deamons with 3 terminals.
>
> % bin/hama zookeeper
> % bin/hama bspmaster
> % bin/hama groom
>
> 3. Use the logging API to log debug.
> For example, LOG.debug("like this");
>
> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
> wrote:
> > Hi,
> >
> > I was wondering if anyone can give me instructions on how to debug code
> in
> > distributed or pseudo-distributed mode ? It would really help me.
> >
> > Thank you
> >
> > Sincerely,
> > Ashish
> >
> >
> >
> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
> >wrote:
> >
> >>
> >> Hi,
> >>
> >> I have written some code for my GSoC task (Hama-367) and am now trying
> to
> >> debug.
> >>
> >> I am trying to set up the environment into pseudo-distributed mode so I
> can
> >> debug. I think the value of bsp.master.address in hama-site.xml has to
> be
> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
> >> server), but where do I specify that in the code in eclipse ?
> >>
> >> My next question is - what is the difference between MiniBSPCluster and
> >> LocalBSPRunner  and which file should I be using to be able to step
> through
> >> the code ?
> >>
> >> These are the processes currently running on my machine -
> >>
> >> 6112 NameNode
> >> 3319 ZooKeeperRunner
> >> 3018
> >> 6449 SecondaryNameNode
> >> 5709 HMaster
> >> 5643 HQuorumPeer
> >> 6684 TaskTracker
> >> 6526 JobTracker
> >> 8074 Jps
> >> 6277 DataNode
> >>
> >> Thank you,
> >> Ashish
> >>
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>



-- 
Thomas Jungblut
Berlin

mobile: 0170-3081070

business: thomas.jungblut@testberichte.de
private: thomas.jungblut@gmail.com

Re: Debugging hama in eclipse

Posted by Ashish Agarwal <as...@gmail.com>.

Hi,

I think the error might be because I have to start the BSP Master
server before trying to run the BSP Master node.

But, when I run the command bin/start-bspd.sh, hama tries to use
openjdk for some reason. I have set the default version to sun JDK and
my JAVA_HOME points to java-6-sun.

Any idea why I might be getting this error ?

ashish@ubuntu:~/Desktop/Project/hama-trunk$ bin/start-bspd.sh
localhost: starting zookeeper, logging to
/home/ashish/Desktop/Project/hama-trunk/bin/../logs/hama-ashish-zookeeper-ubuntu.out
localhost: /home/ashish/Desktop/Project/hama-trunk/bin/../bin/hama:
line 186: /home/ashish/Desktop/Project/hama-trunk/”/usr/lib/jvm/java-6-openjdk”/bin/java:
No such file or directory
starting bspmaster, logging to
/home/ashish/Desktop/Project/hama-trunk/bin/../logs/hama-ashish-bspmaster-ubuntu.out
localhost: starting groom, logging to
/home/ashish/Desktop/Project/hama-trunk/bin/../logs/hama-ashish-groom-ubuntu.out
localhost: /home/ashish/Desktop/Project/hama-trunk/bin/../bin/hama:
line 186: /home/ashish/Desktop/Project/hama-trunk/”/usr/lib/jvm/java-6-openjdk”/bin/java:
No such file or directory
ashish@ubuntu:~/Desktop/Project/hama-trunk$


Thanks
Ashish


On Thu, Jun 23, 2011 at 2:49 AM, Edward J. Yoon <ed...@apache.org> wrote:
> Hey,
>
> Let's see the error message.
>
>> java.net.UnknownHostException: Invalid hostname for server: local
>
> and
>
>> at org.apache.hama.bsp.BSPMaster.<init>(BSPMaster.java:248)
>
> line 248:   host = getAddress(conf).getHostName();
>
> I think, you have to try to fix/debug your problem yourself before calling.
>
> On Thu, Jun 23, 2011 at 6:33 PM, Ashish Agarwal <as...@gmail.com> wrote:
>> It didn't work.
>> I have attached my  hama-site.xml.template file.
>>
>>
>> On Thu, Jun 23, 2011 at 2:16 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>> Have you tried to run with <value>ubuntu</value>?
>>>
>>> On Thu, Jun 23, 2011 at 6:04 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>>> <property>
>>>>   <name>fs.default.name</name>
>>>>    <value>localhost:9000</value>
>>>>  </property>
>>>>
>>>> This is the same in both hadoop-site.xml and the hama-site.xml
>>>>
>>>> The /etc/hosts file looks like this -
>>>>
>>>> 69.90.54.2      ubuntu  # Added by NetworkManager
>>>> 127.0.0.1       localhost.localdomain   localhost
>>>> ::1     ubuntu  localhost6.localdomain6 localhost6
>>>> 127.0.1.1       ubuntu.ubuntu-domain    ubuntu
>>>>
>>>> # The following lines are desirable for IPv6 capable hosts
>>>> ::1     localhost ip6-localhost ip6-loopback
>>>> fe00::0 ip6-localnet
>>>> ff00::0 ip6-mcastprefix
>>>> ff02::1 ip6-allnodes
>>>> ff02::2 ip6-allrouters
>>>> ff02::3 ip6-allhosts
>>>>
>>>> Thanks
>>>> Ashish
>>>>
>>>>
>>>> On Thu, Jun 23, 2011 at 1:40 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>> What is your 'fs.default.name' of hadoop?
>>>>>
>>>>> What is your hostname?
>>>>>
>>>>> Have you seen the /etc/hosts file?
>>>>>
>>>>> On Thu, Jun 23, 2011 at 5:22 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>>>>> I already have made that change, but I still have the same error
>>>>>>
>>>>>> I have tried
>>>>>> <value>localhost:40000</value>
>>>>>> and also
>>>>>> <value>local</value>
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Ashish
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 23, 2011 at 1:12 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>>>>
>>>>>>> You need to set the 'bsp.master.address' to localhost or your hostname.
>>>>>>>
>>>>>>> conf/hama-site.xml:
>>>>>>>
>>>>>>>  <property>
>>>>>>>    <name>bsp.master.address</name>
>>>>>>>    <value>localhost:40000</value>
>>>>>>>  </property>
>>>>>>>
>>>>>>> On Thu, Jun 23, 2011 at 5:01 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>>>>>> > Now, I am having a problem starting the bspmaster :(
>>>>>>> > bin/hama bspmaster
>>>>>>> >
>>>>>>> >
>>>>>>> > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama bspmaster
>>>>>>> > 11/06/23 01:00:13 INFO hama.BSPMasterRunner: STARTUP_MSG:
>>>>>>> > /************************************************************
>>>>>>> > STARTUP_MSG: Starting BSPMaster
>>>>>>> > STARTUP_MSG:   host = ubuntu/127.0.1.1
>>>>>>> > STARTUP_MSG:   args = []
>>>>>>> > STARTUP_MSG:   version = 0.20.2
>>>>>>> > STARTUP_MSG:   build =  -r ; compiled by 'chrisdo' on Wed Mar 10
>>>>>>> > 13:22:54 PST 2010
>>>>>>> > ************************************************************/
>>>>>>> > 11/06/23 01:00:13 INFO bsp.BSPMaster: RPC BSPMaster: host local port 40000
>>>>>>> > 11/06/23 01:00:14 FATAL hama.BSPMasterRunner:
>>>>>>> > java.net.UnknownHostException: Invalid hostname for server: local
>>>>>>> >        at org.apache.hadoop.ipc.Server.bind(Server.java:198)
>>>>>>> >        at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
>>>>>>> >        at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
>>>>>>> >        at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
>>>>>>> >        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
>>>>>>> >        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:441)
>>>>>>> >        at org.apache.hama.bsp.BSPMaster.<init>(BSPMaster.java:248)
>>>>>>> >        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:419)
>>>>>>> >        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:413)
>>>>>>> >        at org.apache.hama.BSPMasterRunner.run(BSPMasterRunner.java:46)
>>>>>>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>>> >        at org.apache.hama.BSPMasterRunner.main(BSPMasterRunner.java:56)
>>>>>>> >
>>>>>>> > 11/06/23 01:00:14 INFO hama.BSPMasterRunner: SHUTDOWN_MSG:
>>>>>>> > /************************************************************
>>>>>>> > SHUTDOWN_MSG: Shutting down BSPMaster at ubuntu/127.0.1.1
>>>>>>> > ************************************************************/
>>>>>>> > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>>>>>>> >
>>>>>>> >
>>>>>>> > Thanks
>>>>>>> > Ashish
>>>>>>> >
>>>>>>> > On Thu, Jun 23, 2011 at 12:49 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>>>> >> If you want to execute a command in the background without watching
>>>>>>> >> logs, you can do the following:
>>>>>>> >>
>>>>>>> >> % bin/hama zookeeper &
>>>>>>> >>
>>>>>>> >> On Thu, Jun 23, 2011 at 4:43 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>>>>>> >>> How long does it take to execute the command
>>>>>>> >>> bin/hama zookeeper
>>>>>>> >>>
>>>>>>> >>> I had to stop the execution as it stopped doing anything after some time.
>>>>>>> >>>
>>>>>>> >>> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>>> >>> environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
>>>>>>> >>> GMT
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>>> >>> environment:host.name=ubuntu.ubuntu-domain
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>>> >>> environment:java.version=1.6.0_24
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>>> >>> environment:java.vendor=Sun Microsystems Inc.
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>>> >>> environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>>> >>> environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>>> >>> environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>>> >>> environment:java.io.tmpdir=/tmp
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>>> >>> environment:java.compiler=<NA>
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.name=Linux
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.arch=i386
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>>> >>> environment:os.version=2.6.35-28-generic
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>>> >>> environment:user.name=hadoop
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>>> >>> environment:user.home=/home/hadoop
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>>> >>> environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1
>>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to 1200000
>>>>>>> >>> 11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port
>>>>>>> >>> 0.0.0.0/0.0.0.0:21810
>>>>>>> >>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>>>>>>> >>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>>>>>>> >>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>>>>>>> >>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>>>>>>> >>> 11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0
>>>>>>> >>> ^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> Thanks
>>>>>>> >>> Ashish
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd
>>>>>>> >>> <jo...@cbsinteractive.com> wrote:
>>>>>>> >>>>
>>>>>>> >>>> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>>>> >>>> > Instead removing 206 line, why don't we add dependencies as below?
>>>>>>> >>>>
>>>>>>> >>>> I think removing line 206 is the right thing to do.  Hama doesn't
>>>>>>> >>>> depend on commons-cli, hadoop-core does, and maven will properly pick
>>>>>>> >>>> that up and include the jar if allowed.
>>>>>>> >>>>
>>>>>>> >>>> The fact that Hama doesn't depend on the several items below was
>>>>>>> >>>> correctly detected by maven, in the report description of HAMA-402.
>>>>>>> >>>>
>>>>>>> >>>> You don't mention a particular problem with including the transitive
>>>>>>> >>>> dependencies, but if your problem is that it includes too many items,
>>>>>>> >>>> maven does have options to exclude a particular transitive dependency
>>>>>>> >>>> on a case-by-case basis that might be more appropriate to use.
>>>>>>> >>>>
>>>>>>> >>>> Including the dependencies below as you suggest puts Hama in the place
>>>>>>> >>>> of maintaining a transitive dependency like commons-cli in lock-step
>>>>>>> >>>> hadoop-core, where the actual dependency comes from.
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>> This is all just my $.02 worth, I'm open to arguments in favor of
>>>>>>> >>>> <excludeTransitive/>, and there is often more than one way to skin a
>>>>>>> >>>> cat with respect to dependency management.
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>> ...joe
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>> >    <dependency>
>>>>>>> >>>> >      <groupId>org.mortbay.jetty</groupId>
>>>>>>> >>>> >      <artifactId>jetty-annotations</artifactId>
>>>>>>> >>>> >      <version>6.1H.22</version>
>>>>>>> >>>> >    </dependency>
>>>>>>> >>>> >    <dependency>
>>>>>>> >>>> >      <groupId>org.slf4j</groupId>
>>>>>>> >>>> >      <artifactId>slf4j-log4j12</artifactId>
>>>>>>> >>>> >      <version>1.5.2</version>
>>>>>>> >>>> >    </dependency>
>>>>>>> >>>> >    <dependency>
>>>>>>> >>>> >      <groupId>commons-cli</groupId>
>>>>>>> >>>> >      <artifactId>commons-cli</artifactId>
>>>>>>> >>>> >      <version>1.2</version>
>>>>>>> >>>> >    </dependency>
>>>>>>> >>>> >
>>>>>>> >>>> > Thanks.
>>>>>>> >>>> >
>>>>>>> >>>> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>>>> >>>> >> Oh... sorry, you're right.
>>>>>>> >>>> >>
>>>>>>> >>>> >> It looks like a bug. We have to fix this.
>>>>>>> >>>> >>
>>>>>>> >>>> >> [INFO] Total time: 4:36.098s
>>>>>>> >>>> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
>>>>>>> >>>> >> [INFO] Final Memory: 20M/51M
>>>>>>> >>>> >> [INFO] ------------------------------------------------------------------------
>>>>>>> >>>> >> edward@test ~/workspace/hama-trunk $ ls lib/
>>>>>>> >>>> >> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>>>>>> >>>> >> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>>>>>> >>>> >> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>>>>>> >>>> >>    zookeeper-3.3.1.jar
>>>>>>> >>>> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
>>>>>>> >>>> >> -bash: bin/hama: Permission denied
>>>>>>> >>>> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
>>>>>>> >>>> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
>>>>>>> >>>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>>> >>>> >> org/apache/commons/cli/ParseException
>>>>>>> >>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>>>>> >>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>>> >>>> >>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>>>>> >>>> >> Caused by: java.lang.ClassNotFoundException:
>>>>>>> >>>> >> org.apache.commons.cli.ParseException
>>>>>>> >>>> >>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>>>>> >>>> >>        at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> >>>> >>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>>>>> >>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
>>>>>>> >>>> >>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>>> >>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>>>>> >>>> >>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
>>>>>>> >>>> >>        ... 3 more
>>>>>>> >>>> >>
>>>>>>> >>>> >>
>>>>>>> >>>> >>
>>>>>>> >>>> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>>>> >>>> >>> That's so weird.
>>>>>>> >>>> >>>
>>>>>>> >>>> >>> It works well on my test cluster.
>>>>>>> >>>> >>>
>>>>>>> >>>> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
>>>>>>> >>>> >>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>>>>>> >>>> >>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>>>>>> >>>> >>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>>>>>> >>>> >>>    zookeeper-3.3.1.jar
>>>>>>> >>>> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
>>>>>>> >>>> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
>>>>>>> >>>> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
>>>>>>> >>>> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
>>>>>>> >>>> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
>>>>>>> >>>> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
>>>>>>> >>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
>>>>>>> >>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
>>>>>>> >>>> >>> Estimated value of PI is 3.1534
>>>>>>> >>>> >>> Job Finished in 7.886 seconds
>>>>>>> >>>> >>> edward@slave:~/workspace/hama-trunk$
>>>>>>> >>>> >>>
>>>>>>> >>>> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
>>>>>>> >>>> >>> <jo...@cbsinteractive.com> wrote:
>>>>>>> >>>> >>>> Replying to my own post ...
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>>>>>>> >>>> >>>> <jo...@cbsinteractive.com> wrote:
>>>>>>> >>>> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>> When I try to run the hama daemons using the command -
>>>>>>> >>>> >>>>>> bin/hama zookeeper
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>> I get this error -
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>> It looks like the problem is actually this line in pom.xml (line 205)  :
>>>>>>> >>>> >>>>                    <excludeTransitive>true</excludeTransitive>
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>> This prevents commons-cli (and other transitive dependencies) from
>>>>>>> >>>> >>>> being copied to the lib dir like they should be.
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
>>>>>>> >>>> >>>> JIRA for this after lunch, the command line is fairly broken with this
>>>>>>> >>>> >>>> line in, especially after several dependencies were (correctly)
>>>>>>> >>>> >>>> removed in HAMA-402.
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>> ...joe
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
>>>>>>> >>>> >>>>> commons-cli dependency was removed recently as part of HAMA-402, and
>>>>>>> >>>> >>>>> re-adding it fixes the NoClassDefFoundError.
>>>>>>> >>>> >>>>>
>>>>>>> >>>> >>>>> I believe commons-cli should have been pulled as part of the
>>>>>>> >>>> >>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
>>>>>>> >>>> >>>>> rusty, and I'm no longer sure just how transitive dependencies are
>>>>>>> >>>> >>>>> pulled)
>>>>>>> >>>> >>>>>
>>>>>>> >>>> >>>>> While a quick work-around is to re-add the dependency here, the proper
>>>>>>> >>>> >>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
>>>>>>> >>>> >>>>> depends on commons-cli.
>>>>>>> >>>> >>>>>
>>>>>>> >>>> >>>>>
>>>>>>> >>>> >>>>> ...joe
>>>>>>> >>>> >>>>>
>>>>>>> >>>> >>>>>
>>>>>>> >>>> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>>> >>>> >>>>>> org/apache/commons/cli/ParseException
>>>>>>> >>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>>>>> >>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>>> >>>> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>>>>> >>>> >>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>> >>>> >>>>>> org.apache.commons.cli.ParseException
>>>>>>> >>>> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>>> >>>> >>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> >>>> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>>> >>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>>>> >>>> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>>> >>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>>>>> >>>> >>>>>> ... 3 more
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>> I tried installing it a few times, but can't think of a solution to the
>>>>>>> >>>> >>>>>> problem. Can you please help ?
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>> Thank you
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>> Sincerely,
>>>>>>> >>>> >>>>>> Ashish
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>>> Hi,
>>>>>>> >>>> >>>>>>>
>>>>>>> >>>> >>>>>>> I would suggest following steps to debug:
>>>>>>> >>>> >>>>>>>
>>>>>>> >>>> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>>>>>>> >>>> >>>>>>> 2. Run the Hama deamons with 3 terminals.
>>>>>>> >>>> >>>>>>>
>>>>>>> >>>> >>>>>>> % bin/hama zookeeper
>>>>>>> >>>> >>>>>>> % bin/hama bspmaster
>>>>>>> >>>> >>>>>>> % bin/hama groom
>>>>>>> >>>> >>>>>>>
>>>>>>> >>>> >>>>>>> 3. Use the logging API to log debug.
>>>>>>> >>>> >>>>>>> For example, LOG.debug("like this");
>>>>>>> >>>> >>>>>>>
>>>>>>> >>>> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>>>>>>> >>>> >>>>>>> wrote:
>>>>>>> >>>> >>>>>>> > Hi,
>>>>>>> >>>> >>>>>>> >
>>>>>>> >>>> >>>>>>> > I was wondering if anyone can give me instructions on how to debug code
>>>>>>> >>>> >>>>>>> in
>>>>>>> >>>> >>>>>>> > distributed or pseudo-distributed mode ? It would really help me.
>>>>>>> >>>> >>>>>>> >
>>>>>>> >>>> >>>>>>> > Thank you
>>>>>>> >>>> >>>>>>> >
>>>>>>> >>>> >>>>>>> > Sincerely,
>>>>>>> >>>> >>>>>>> > Ashish
>>>>>>> >>>> >>>>>>> >
>>>>>>> >>>> >>>>>>> >
>>>>>>> >>>> >>>>>>> >
>>>>>>> >>>> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>>>>>>> >>>> >>>>>>> >wrote:
>>>>>>> >>>> >>>>>>> >
>>>>>>> >>>> >>>>>>> >>
>>>>>>> >>>> >>>>>>> >> Hi,
>>>>>>> >>>> >>>>>>> >>
>>>>>>> >>>> >>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>>>>>>> >>>> >>>>>>> to
>>>>>>> >>>> >>>>>>> >> debug.
>>>>>>> >>>> >>>>>>> >>
>>>>>>> >>>> >>>>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
>>>>>>> >>>> >>>>>>> can
>>>>>>> >>>> >>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>>>>>>> >>>> >>>>>>> be
>>>>>>> >>>> >>>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>>>>>>> >>>> >>>>>>> >> server), but where do I specify that in the code in eclipse ?
>>>>>>> >>>> >>>>>>> >>
>>>>>>> >>>> >>>>>>> >> My next question is - what is the difference between MiniBSPCluster and
>>>>>>> >>>> >>>>>>> >> LocalBSPRunner  and which file should I be using to be able to step
>>>>>>> >>>> >>>>>>> through
>>>>>>> >>>> >>>>>>> >> the code ?
>>>>>>> >>>> >>>>>>> >>
>>>>>>> >>>> >>>>>>> >> These are the processes currently running on my machine -
>>>>>>> >>>> >>>>>>> >>
>>>>>>> >>>> >>>>>>> >> 6112 NameNode
>>>>>>> >>>> >>>>>>> >> 3319 ZooKeeperRunner
>>>>>>> >>>> >>>>>>> >> 3018
>>>>>>> >>>> >>>>>>> >> 6449 SecondaryNameNode
>>>>>>> >>>> >>>>>>> >> 5709 HMaster
>>>>>>> >>>> >>>>>>> >> 5643 HQuorumPeer
>>>>>>> >>>> >>>>>>> >> 6684 TaskTracker
>>>>>>> >>>> >>>>>>> >> 6526 JobTracker
>>>>>>> >>>> >>>>>>> >> 8074 Jps
>>>>>>> >>>> >>>>>>> >> 6277 DataNode
>>>>>>> >>>> >>>>>>> >>
>>>>>>> >>>> >>>>>>> >> Thank you,
>>>>>>> >>>> >>>>>>> >> Ashish
>>>>>>> >>>> >>>>>>> >>
>>>>>>> >>>> >>>>>>> >
>>>>>>> >>>> >>>>>>>
>>>>>>> >>>> >>>>>>>
>>>>>>> >>>> >>>>>>>
>>>>>>> >>>> >>>>>>> --
>>>>>>> >>>> >>>>>>> Best Regards, Edward J. Yoon
>>>>>>> >>>> >>>>>>> @eddieyoon
>>>>>>> >>>> >>>>>>>
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>
>>>>>>> >>>> >>>
>>>>>>> >>>> >>>
>>>>>>> >>>> >>> --
>>>>>>> >>>> >>> Best Regards, Edward J. Yoon
>>>>>>> >>>> >>> @eddieyoon
>>>>>>> >>>> >>>
>>>>>>> >>>> >>
>>>>>>> >>>> >>
>>>>>>> >>>> >>
>>>>>>> >>>> >> --
>>>>>>> >>>> >> Best Regards, Edward J. Yoon
>>>>>>> >>>> >> @eddieyoon
>>>>>>> >>>> >>
>>>>>>> >>>> >
>>>>>>> >>>> >
>>>>>>> >>>> >
>>>>>>> >>>> > --
>>>>>>> >>>> > Best Regards, Edward J. Yoon
>>>>>>> >>>> > @eddieyoon
>>>>>>> >>>> >
>>>>>>> >>>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> --
>>>>>>> >> Best Regards, Edward J. Yoon
>>>>>>> >> @eddieyoon
>>>>>>> >>
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards, Edward J. Yoon
>>>>>>> @eddieyoon
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards, Edward J. Yoon
>>>>> @eddieyoon
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>>
>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>

Re: Debugging hama in eclipse

Posted by "Edward J. Yoon" <ed...@apache.org>.

Hey,

Let's see the error message.

> java.net.UnknownHostException: Invalid hostname for server: local

and

> at org.apache.hama.bsp.BSPMaster.<init>(BSPMaster.java:248)

line 248:   host = getAddress(conf).getHostName();

I think, you have to try to fix/debug your problem yourself before calling.

On Thu, Jun 23, 2011 at 6:33 PM, Ashish Agarwal <as...@gmail.com> wrote:
> It didn't work.
> I have attached my  hama-site.xml.template file.
>
>
> On Thu, Jun 23, 2011 at 2:16 AM, Edward J. Yoon <ed...@apache.org> wrote:
>> Have you tried to run with <value>ubuntu</value>?
>>
>> On Thu, Jun 23, 2011 at 6:04 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>> <property>
>>>   <name>fs.default.name</name>
>>>    <value>localhost:9000</value>
>>>  </property>
>>>
>>> This is the same in both hadoop-site.xml and the hama-site.xml
>>>
>>> The /etc/hosts file looks like this -
>>>
>>> 69.90.54.2      ubuntu  # Added by NetworkManager
>>> 127.0.0.1       localhost.localdomain   localhost
>>> ::1     ubuntu  localhost6.localdomain6 localhost6
>>> 127.0.1.1       ubuntu.ubuntu-domain    ubuntu
>>>
>>> # The following lines are desirable for IPv6 capable hosts
>>> ::1     localhost ip6-localhost ip6-loopback
>>> fe00::0 ip6-localnet
>>> ff00::0 ip6-mcastprefix
>>> ff02::1 ip6-allnodes
>>> ff02::2 ip6-allrouters
>>> ff02::3 ip6-allhosts
>>>
>>> Thanks
>>> Ashish
>>>
>>>
>>> On Thu, Jun 23, 2011 at 1:40 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>> What is your 'fs.default.name' of hadoop?
>>>>
>>>> What is your hostname?
>>>>
>>>> Have you seen the /etc/hosts file?
>>>>
>>>> On Thu, Jun 23, 2011 at 5:22 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>>>> I already have made that change, but I still have the same error
>>>>>
>>>>> I have tried
>>>>> <value>localhost:40000</value>
>>>>> and also
>>>>> <value>local</value>
>>>>>
>>>>>
>>>>> Thanks
>>>>> Ashish
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 23, 2011 at 1:12 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>>>
>>>>>> You need to set the 'bsp.master.address' to localhost or your hostname.
>>>>>>
>>>>>> conf/hama-site.xml:
>>>>>>
>>>>>>  <property>
>>>>>>    <name>bsp.master.address</name>
>>>>>>    <value>localhost:40000</value>
>>>>>>  </property>
>>>>>>
>>>>>> On Thu, Jun 23, 2011 at 5:01 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>>>>> > Now, I am having a problem starting the bspmaster :(
>>>>>> > bin/hama bspmaster
>>>>>> >
>>>>>> >
>>>>>> > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama bspmaster
>>>>>> > 11/06/23 01:00:13 INFO hama.BSPMasterRunner: STARTUP_MSG:
>>>>>> > /************************************************************
>>>>>> > STARTUP_MSG: Starting BSPMaster
>>>>>> > STARTUP_MSG:   host = ubuntu/127.0.1.1
>>>>>> > STARTUP_MSG:   args = []
>>>>>> > STARTUP_MSG:   version = 0.20.2
>>>>>> > STARTUP_MSG:   build =  -r ; compiled by 'chrisdo' on Wed Mar 10
>>>>>> > 13:22:54 PST 2010
>>>>>> > ************************************************************/
>>>>>> > 11/06/23 01:00:13 INFO bsp.BSPMaster: RPC BSPMaster: host local port 40000
>>>>>> > 11/06/23 01:00:14 FATAL hama.BSPMasterRunner:
>>>>>> > java.net.UnknownHostException: Invalid hostname for server: local
>>>>>> >        at org.apache.hadoop.ipc.Server.bind(Server.java:198)
>>>>>> >        at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
>>>>>> >        at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
>>>>>> >        at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
>>>>>> >        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
>>>>>> >        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:441)
>>>>>> >        at org.apache.hama.bsp.BSPMaster.<init>(BSPMaster.java:248)
>>>>>> >        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:419)
>>>>>> >        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:413)
>>>>>> >        at org.apache.hama.BSPMasterRunner.run(BSPMasterRunner.java:46)
>>>>>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>> >        at org.apache.hama.BSPMasterRunner.main(BSPMasterRunner.java:56)
>>>>>> >
>>>>>> > 11/06/23 01:00:14 INFO hama.BSPMasterRunner: SHUTDOWN_MSG:
>>>>>> > /************************************************************
>>>>>> > SHUTDOWN_MSG: Shutting down BSPMaster at ubuntu/127.0.1.1
>>>>>> > ************************************************************/
>>>>>> > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>>>>>> >
>>>>>> >
>>>>>> > Thanks
>>>>>> > Ashish
>>>>>> >
>>>>>> > On Thu, Jun 23, 2011 at 12:49 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>>> >> If you want to execute a command in the background without watching
>>>>>> >> logs, you can do the following:
>>>>>> >>
>>>>>> >> % bin/hama zookeeper &
>>>>>> >>
>>>>>> >> On Thu, Jun 23, 2011 at 4:43 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>>>>> >>> How long does it take to execute the command
>>>>>> >>> bin/hama zookeeper
>>>>>> >>>
>>>>>> >>> I had to stop the execution as it stopped doing anything after some time.
>>>>>> >>>
>>>>>> >>> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>> >>> environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
>>>>>> >>> GMT
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>> >>> environment:host.name=ubuntu.ubuntu-domain
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>> >>> environment:java.version=1.6.0_24
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>> >>> environment:java.vendor=Sun Microsystems Inc.
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>> >>> environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>> >>> environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>> >>> environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>> >>> environment:java.io.tmpdir=/tmp
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>> >>> environment:java.compiler=<NA>
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.name=Linux
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.arch=i386
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>> >>> environment:os.version=2.6.35-28-generic
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>> >>> environment:user.name=hadoop
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>> >>> environment:user.home=/home/hadoop
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>>> >>> environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1
>>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to 1200000
>>>>>> >>> 11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port
>>>>>> >>> 0.0.0.0/0.0.0.0:21810
>>>>>> >>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>>>>>> >>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>>>>>> >>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>>>>>> >>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>>>>>> >>> 11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0
>>>>>> >>> ^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> Thanks
>>>>>> >>> Ashish
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd
>>>>>> >>> <jo...@cbsinteractive.com> wrote:
>>>>>> >>>>
>>>>>> >>>> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>>> >>>> > Instead removing 206 line, why don't we add dependencies as below?
>>>>>> >>>>
>>>>>> >>>> I think removing line 206 is the right thing to do.  Hama doesn't
>>>>>> >>>> depend on commons-cli, hadoop-core does, and maven will properly pick
>>>>>> >>>> that up and include the jar if allowed.
>>>>>> >>>>
>>>>>> >>>> The fact that Hama doesn't depend on the several items below was
>>>>>> >>>> correctly detected by maven, in the report description of HAMA-402.
>>>>>> >>>>
>>>>>> >>>> You don't mention a particular problem with including the transitive
>>>>>> >>>> dependencies, but if your problem is that it includes too many items,
>>>>>> >>>> maven does have options to exclude a particular transitive dependency
>>>>>> >>>> on a case-by-case basis that might be more appropriate to use.
>>>>>> >>>>
>>>>>> >>>> Including the dependencies below as you suggest puts Hama in the place
>>>>>> >>>> of maintaining a transitive dependency like commons-cli in lock-step
>>>>>> >>>> hadoop-core, where the actual dependency comes from.
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> This is all just my $.02 worth, I'm open to arguments in favor of
>>>>>> >>>> <excludeTransitive/>, and there is often more than one way to skin a
>>>>>> >>>> cat with respect to dependency management.
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> ...joe
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> >    <dependency>
>>>>>> >>>> >      <groupId>org.mortbay.jetty</groupId>
>>>>>> >>>> >      <artifactId>jetty-annotations</artifactId>
>>>>>> >>>> >      <version>6.1H.22</version>
>>>>>> >>>> >    </dependency>
>>>>>> >>>> >    <dependency>
>>>>>> >>>> >      <groupId>org.slf4j</groupId>
>>>>>> >>>> >      <artifactId>slf4j-log4j12</artifactId>
>>>>>> >>>> >      <version>1.5.2</version>
>>>>>> >>>> >    </dependency>
>>>>>> >>>> >    <dependency>
>>>>>> >>>> >      <groupId>commons-cli</groupId>
>>>>>> >>>> >      <artifactId>commons-cli</artifactId>
>>>>>> >>>> >      <version>1.2</version>
>>>>>> >>>> >    </dependency>
>>>>>> >>>> >
>>>>>> >>>> > Thanks.
>>>>>> >>>> >
>>>>>> >>>> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>>> >>>> >> Oh... sorry, you're right.
>>>>>> >>>> >>
>>>>>> >>>> >> It looks like a bug. We have to fix this.
>>>>>> >>>> >>
>>>>>> >>>> >> [INFO] Total time: 4:36.098s
>>>>>> >>>> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
>>>>>> >>>> >> [INFO] Final Memory: 20M/51M
>>>>>> >>>> >> [INFO] ------------------------------------------------------------------------
>>>>>> >>>> >> edward@test ~/workspace/hama-trunk $ ls lib/
>>>>>> >>>> >> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>>>>> >>>> >> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>>>>> >>>> >> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>>>>> >>>> >>    zookeeper-3.3.1.jar
>>>>>> >>>> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
>>>>>> >>>> >> -bash: bin/hama: Permission denied
>>>>>> >>>> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
>>>>>> >>>> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
>>>>>> >>>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>> >>>> >> org/apache/commons/cli/ParseException
>>>>>> >>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>>>> >>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>> >>>> >>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>>>> >>>> >> Caused by: java.lang.ClassNotFoundException:
>>>>>> >>>> >> org.apache.commons.cli.ParseException
>>>>>> >>>> >>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>>>> >>>> >>        at java.security.AccessController.doPrivileged(Native Method)
>>>>>> >>>> >>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>>>> >>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
>>>>>> >>>> >>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>> >>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>>>> >>>> >>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
>>>>>> >>>> >>        ... 3 more
>>>>>> >>>> >>
>>>>>> >>>> >>
>>>>>> >>>> >>
>>>>>> >>>> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>>> >>>> >>> That's so weird.
>>>>>> >>>> >>>
>>>>>> >>>> >>> It works well on my test cluster.
>>>>>> >>>> >>>
>>>>>> >>>> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
>>>>>> >>>> >>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>>>>> >>>> >>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>>>>> >>>> >>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>>>>> >>>> >>>    zookeeper-3.3.1.jar
>>>>>> >>>> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
>>>>>> >>>> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
>>>>>> >>>> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
>>>>>> >>>> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
>>>>>> >>>> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
>>>>>> >>>> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
>>>>>> >>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
>>>>>> >>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
>>>>>> >>>> >>> Estimated value of PI is 3.1534
>>>>>> >>>> >>> Job Finished in 7.886 seconds
>>>>>> >>>> >>> edward@slave:~/workspace/hama-trunk$
>>>>>> >>>> >>>
>>>>>> >>>> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
>>>>>> >>>> >>> <jo...@cbsinteractive.com> wrote:
>>>>>> >>>> >>>> Replying to my own post ...
>>>>>> >>>> >>>>
>>>>>> >>>> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>>>>>> >>>> >>>> <jo...@cbsinteractive.com> wrote:
>>>>>> >>>> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>> When I try to run the hama daemons using the command -
>>>>>> >>>> >>>>>> bin/hama zookeeper
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>> I get this error -
>>>>>> >>>> >>>>
>>>>>> >>>> >>>> It looks like the problem is actually this line in pom.xml (line 205)  :
>>>>>> >>>> >>>>                    <excludeTransitive>true</excludeTransitive>
>>>>>> >>>> >>>>
>>>>>> >>>> >>>> This prevents commons-cli (and other transitive dependencies) from
>>>>>> >>>> >>>> being copied to the lib dir like they should be.
>>>>>> >>>> >>>>
>>>>>> >>>> >>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
>>>>>> >>>> >>>> JIRA for this after lunch, the command line is fairly broken with this
>>>>>> >>>> >>>> line in, especially after several dependencies were (correctly)
>>>>>> >>>> >>>> removed in HAMA-402.
>>>>>> >>>> >>>>
>>>>>> >>>> >>>>
>>>>>> >>>> >>>> ...joe
>>>>>> >>>> >>>>
>>>>>> >>>> >>>>
>>>>>> >>>> >>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
>>>>>> >>>> >>>>> commons-cli dependency was removed recently as part of HAMA-402, and
>>>>>> >>>> >>>>> re-adding it fixes the NoClassDefFoundError.
>>>>>> >>>> >>>>>
>>>>>> >>>> >>>>> I believe commons-cli should have been pulled as part of the
>>>>>> >>>> >>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
>>>>>> >>>> >>>>> rusty, and I'm no longer sure just how transitive dependencies are
>>>>>> >>>> >>>>> pulled)
>>>>>> >>>> >>>>>
>>>>>> >>>> >>>>> While a quick work-around is to re-add the dependency here, the proper
>>>>>> >>>> >>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
>>>>>> >>>> >>>>> depends on commons-cli.
>>>>>> >>>> >>>>>
>>>>>> >>>> >>>>>
>>>>>> >>>> >>>>> ...joe
>>>>>> >>>> >>>>>
>>>>>> >>>> >>>>>
>>>>>> >>>> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>> >>>> >>>>>> org/apache/commons/cli/ParseException
>>>>>> >>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>>>> >>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>> >>>> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>>>> >>>> >>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>> >>>> >>>>>> org.apache.commons.cli.ParseException
>>>>>> >>>> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>> >>>> >>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> >>>> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>> >>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>>> >>>> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>> >>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>>>> >>>> >>>>>> ... 3 more
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>> I tried installing it a few times, but can't think of a solution to the
>>>>>> >>>> >>>>>> problem. Can you please help ?
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>> Thank you
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>> Sincerely,
>>>>>> >>>> >>>>>> Ashish
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>>> Hi,
>>>>>> >>>> >>>>>>>
>>>>>> >>>> >>>>>>> I would suggest following steps to debug:
>>>>>> >>>> >>>>>>>
>>>>>> >>>> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>>>>>> >>>> >>>>>>> 2. Run the Hama deamons with 3 terminals.
>>>>>> >>>> >>>>>>>
>>>>>> >>>> >>>>>>> % bin/hama zookeeper
>>>>>> >>>> >>>>>>> % bin/hama bspmaster
>>>>>> >>>> >>>>>>> % bin/hama groom
>>>>>> >>>> >>>>>>>
>>>>>> >>>> >>>>>>> 3. Use the logging API to log debug.
>>>>>> >>>> >>>>>>> For example, LOG.debug("like this");
>>>>>> >>>> >>>>>>>
>>>>>> >>>> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>>>>>> >>>> >>>>>>> wrote:
>>>>>> >>>> >>>>>>> > Hi,
>>>>>> >>>> >>>>>>> >
>>>>>> >>>> >>>>>>> > I was wondering if anyone can give me instructions on how to debug code
>>>>>> >>>> >>>>>>> in
>>>>>> >>>> >>>>>>> > distributed or pseudo-distributed mode ? It would really help me.
>>>>>> >>>> >>>>>>> >
>>>>>> >>>> >>>>>>> > Thank you
>>>>>> >>>> >>>>>>> >
>>>>>> >>>> >>>>>>> > Sincerely,
>>>>>> >>>> >>>>>>> > Ashish
>>>>>> >>>> >>>>>>> >
>>>>>> >>>> >>>>>>> >
>>>>>> >>>> >>>>>>> >
>>>>>> >>>> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>>>>>> >>>> >>>>>>> >wrote:
>>>>>> >>>> >>>>>>> >
>>>>>> >>>> >>>>>>> >>
>>>>>> >>>> >>>>>>> >> Hi,
>>>>>> >>>> >>>>>>> >>
>>>>>> >>>> >>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>>>>>> >>>> >>>>>>> to
>>>>>> >>>> >>>>>>> >> debug.
>>>>>> >>>> >>>>>>> >>
>>>>>> >>>> >>>>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
>>>>>> >>>> >>>>>>> can
>>>>>> >>>> >>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>>>>>> >>>> >>>>>>> be
>>>>>> >>>> >>>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>>>>>> >>>> >>>>>>> >> server), but where do I specify that in the code in eclipse ?
>>>>>> >>>> >>>>>>> >>
>>>>>> >>>> >>>>>>> >> My next question is - what is the difference between MiniBSPCluster and
>>>>>> >>>> >>>>>>> >> LocalBSPRunner  and which file should I be using to be able to step
>>>>>> >>>> >>>>>>> through
>>>>>> >>>> >>>>>>> >> the code ?
>>>>>> >>>> >>>>>>> >>
>>>>>> >>>> >>>>>>> >> These are the processes currently running on my machine -
>>>>>> >>>> >>>>>>> >>
>>>>>> >>>> >>>>>>> >> 6112 NameNode
>>>>>> >>>> >>>>>>> >> 3319 ZooKeeperRunner
>>>>>> >>>> >>>>>>> >> 3018
>>>>>> >>>> >>>>>>> >> 6449 SecondaryNameNode
>>>>>> >>>> >>>>>>> >> 5709 HMaster
>>>>>> >>>> >>>>>>> >> 5643 HQuorumPeer
>>>>>> >>>> >>>>>>> >> 6684 TaskTracker
>>>>>> >>>> >>>>>>> >> 6526 JobTracker
>>>>>> >>>> >>>>>>> >> 8074 Jps
>>>>>> >>>> >>>>>>> >> 6277 DataNode
>>>>>> >>>> >>>>>>> >>
>>>>>> >>>> >>>>>>> >> Thank you,
>>>>>> >>>> >>>>>>> >> Ashish
>>>>>> >>>> >>>>>>> >>
>>>>>> >>>> >>>>>>> >
>>>>>> >>>> >>>>>>>
>>>>>> >>>> >>>>>>>
>>>>>> >>>> >>>>>>>
>>>>>> >>>> >>>>>>> --
>>>>>> >>>> >>>>>>> Best Regards, Edward J. Yoon
>>>>>> >>>> >>>>>>> @eddieyoon
>>>>>> >>>> >>>>>>>
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>
>>>>>> >>>> >>>>
>>>>>> >>>> >>>
>>>>>> >>>> >>>
>>>>>> >>>> >>>
>>>>>> >>>> >>> --
>>>>>> >>>> >>> Best Regards, Edward J. Yoon
>>>>>> >>>> >>> @eddieyoon
>>>>>> >>>> >>>
>>>>>> >>>> >>
>>>>>> >>>> >>
>>>>>> >>>> >>
>>>>>> >>>> >> --
>>>>>> >>>> >> Best Regards, Edward J. Yoon
>>>>>> >>>> >> @eddieyoon
>>>>>> >>>> >>
>>>>>> >>>> >
>>>>>> >>>> >
>>>>>> >>>> >
>>>>>> >>>> > --
>>>>>> >>>> > Best Regards, Edward J. Yoon
>>>>>> >>>> > @eddieyoon
>>>>>> >>>> >
>>>>>> >>>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> Best Regards, Edward J. Yoon
>>>>>> >> @eddieyoon
>>>>>> >>
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards, Edward J. Yoon
>>>>>> @eddieyoon
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards, Edward J. Yoon
>>>> @eddieyoon
>>>>
>>>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: Debugging hama in eclipse

Posted by Ashish Agarwal <as...@gmail.com>.

It didn't work.
I have attached my  hama-site.xml.template file.


On Thu, Jun 23, 2011 at 2:16 AM, Edward J. Yoon <ed...@apache.org> wrote:
> Have you tried to run with <value>ubuntu</value>?
>
> On Thu, Jun 23, 2011 at 6:04 PM, Ashish Agarwal <as...@gmail.com> wrote:
>> <property>
>>   <name>fs.default.name</name>
>>    <value>localhost:9000</value>
>>  </property>
>>
>> This is the same in both hadoop-site.xml and the hama-site.xml
>>
>> The /etc/hosts file looks like this -
>>
>> 69.90.54.2      ubuntu  # Added by NetworkManager
>> 127.0.0.1       localhost.localdomain   localhost
>> ::1     ubuntu  localhost6.localdomain6 localhost6
>> 127.0.1.1       ubuntu.ubuntu-domain    ubuntu
>>
>> # The following lines are desirable for IPv6 capable hosts
>> ::1     localhost ip6-localhost ip6-loopback
>> fe00::0 ip6-localnet
>> ff00::0 ip6-mcastprefix
>> ff02::1 ip6-allnodes
>> ff02::2 ip6-allrouters
>> ff02::3 ip6-allhosts
>>
>> Thanks
>> Ashish
>>
>>
>> On Thu, Jun 23, 2011 at 1:40 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>> What is your 'fs.default.name' of hadoop?
>>>
>>> What is your hostname?
>>>
>>> Have you seen the /etc/hosts file?
>>>
>>> On Thu, Jun 23, 2011 at 5:22 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>>> I already have made that change, but I still have the same error
>>>>
>>>> I have tried
>>>> <value>localhost:40000</value>
>>>> and also
>>>> <value>local</value>
>>>>
>>>>
>>>> Thanks
>>>> Ashish
>>>>
>>>>
>>>>
>>>> On Thu, Jun 23, 2011 at 1:12 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>>
>>>>> You need to set the 'bsp.master.address' to localhost or your hostname.
>>>>>
>>>>> conf/hama-site.xml:
>>>>>
>>>>>  <property>
>>>>>    <name>bsp.master.address</name>
>>>>>    <value>localhost:40000</value>
>>>>>  </property>
>>>>>
>>>>> On Thu, Jun 23, 2011 at 5:01 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>>>> > Now, I am having a problem starting the bspmaster :(
>>>>> > bin/hama bspmaster
>>>>> >
>>>>> >
>>>>> > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama bspmaster
>>>>> > 11/06/23 01:00:13 INFO hama.BSPMasterRunner: STARTUP_MSG:
>>>>> > /************************************************************
>>>>> > STARTUP_MSG: Starting BSPMaster
>>>>> > STARTUP_MSG:   host = ubuntu/127.0.1.1
>>>>> > STARTUP_MSG:   args = []
>>>>> > STARTUP_MSG:   version = 0.20.2
>>>>> > STARTUP_MSG:   build =  -r ; compiled by 'chrisdo' on Wed Mar 10
>>>>> > 13:22:54 PST 2010
>>>>> > ************************************************************/
>>>>> > 11/06/23 01:00:13 INFO bsp.BSPMaster: RPC BSPMaster: host local port 40000
>>>>> > 11/06/23 01:00:14 FATAL hama.BSPMasterRunner:
>>>>> > java.net.UnknownHostException: Invalid hostname for server: local
>>>>> >        at org.apache.hadoop.ipc.Server.bind(Server.java:198)
>>>>> >        at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
>>>>> >        at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
>>>>> >        at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
>>>>> >        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
>>>>> >        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:441)
>>>>> >        at org.apache.hama.bsp.BSPMaster.<init>(BSPMaster.java:248)
>>>>> >        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:419)
>>>>> >        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:413)
>>>>> >        at org.apache.hama.BSPMasterRunner.run(BSPMasterRunner.java:46)
>>>>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>> >        at org.apache.hama.BSPMasterRunner.main(BSPMasterRunner.java:56)
>>>>> >
>>>>> > 11/06/23 01:00:14 INFO hama.BSPMasterRunner: SHUTDOWN_MSG:
>>>>> > /************************************************************
>>>>> > SHUTDOWN_MSG: Shutting down BSPMaster at ubuntu/127.0.1.1
>>>>> > ************************************************************/
>>>>> > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>>>>> >
>>>>> >
>>>>> > Thanks
>>>>> > Ashish
>>>>> >
>>>>> > On Thu, Jun 23, 2011 at 12:49 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>> >> If you want to execute a command in the background without watching
>>>>> >> logs, you can do the following:
>>>>> >>
>>>>> >> % bin/hama zookeeper &
>>>>> >>
>>>>> >> On Thu, Jun 23, 2011 at 4:43 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>>>> >>> How long does it take to execute the command
>>>>> >>> bin/hama zookeeper
>>>>> >>>
>>>>> >>> I had to stop the execution as it stopped doing anything after some time.
>>>>> >>>
>>>>> >>> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>> >>> environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
>>>>> >>> GMT
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>> >>> environment:host.name=ubuntu.ubuntu-domain
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>> >>> environment:java.version=1.6.0_24
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>> >>> environment:java.vendor=Sun Microsystems Inc.
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>> >>> environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>> >>> environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>> >>> environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>> >>> environment:java.io.tmpdir=/tmp
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>> >>> environment:java.compiler=<NA>
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.name=Linux
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.arch=i386
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>> >>> environment:os.version=2.6.35-28-generic
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>> >>> environment:user.name=hadoop
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>> >>> environment:user.home=/home/hadoop
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>>> >>> environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1
>>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to 1200000
>>>>> >>> 11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port
>>>>> >>> 0.0.0.0/0.0.0.0:21810
>>>>> >>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>>>>> >>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>>>>> >>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>>>>> >>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>>>>> >>> 11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0
>>>>> >>> ^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>>>>> >>>
>>>>> >>>
>>>>> >>> Thanks
>>>>> >>> Ashish
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd
>>>>> >>> <jo...@cbsinteractive.com> wrote:
>>>>> >>>>
>>>>> >>>> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>> >>>> > Instead removing 206 line, why don't we add dependencies as below?
>>>>> >>>>
>>>>> >>>> I think removing line 206 is the right thing to do.  Hama doesn't
>>>>> >>>> depend on commons-cli, hadoop-core does, and maven will properly pick
>>>>> >>>> that up and include the jar if allowed.
>>>>> >>>>
>>>>> >>>> The fact that Hama doesn't depend on the several items below was
>>>>> >>>> correctly detected by maven, in the report description of HAMA-402.
>>>>> >>>>
>>>>> >>>> You don't mention a particular problem with including the transitive
>>>>> >>>> dependencies, but if your problem is that it includes too many items,
>>>>> >>>> maven does have options to exclude a particular transitive dependency
>>>>> >>>> on a case-by-case basis that might be more appropriate to use.
>>>>> >>>>
>>>>> >>>> Including the dependencies below as you suggest puts Hama in the place
>>>>> >>>> of maintaining a transitive dependency like commons-cli in lock-step
>>>>> >>>> hadoop-core, where the actual dependency comes from.
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> This is all just my $.02 worth, I'm open to arguments in favor of
>>>>> >>>> <excludeTransitive/>, and there is often more than one way to skin a
>>>>> >>>> cat with respect to dependency management.
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> ...joe
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> >    <dependency>
>>>>> >>>> >      <groupId>org.mortbay.jetty</groupId>
>>>>> >>>> >      <artifactId>jetty-annotations</artifactId>
>>>>> >>>> >      <version>6.1H.22</version>
>>>>> >>>> >    </dependency>
>>>>> >>>> >    <dependency>
>>>>> >>>> >      <groupId>org.slf4j</groupId>
>>>>> >>>> >      <artifactId>slf4j-log4j12</artifactId>
>>>>> >>>> >      <version>1.5.2</version>
>>>>> >>>> >    </dependency>
>>>>> >>>> >    <dependency>
>>>>> >>>> >      <groupId>commons-cli</groupId>
>>>>> >>>> >      <artifactId>commons-cli</artifactId>
>>>>> >>>> >      <version>1.2</version>
>>>>> >>>> >    </dependency>
>>>>> >>>> >
>>>>> >>>> > Thanks.
>>>>> >>>> >
>>>>> >>>> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>> >>>> >> Oh... sorry, you're right.
>>>>> >>>> >>
>>>>> >>>> >> It looks like a bug. We have to fix this.
>>>>> >>>> >>
>>>>> >>>> >> [INFO] Total time: 4:36.098s
>>>>> >>>> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
>>>>> >>>> >> [INFO] Final Memory: 20M/51M
>>>>> >>>> >> [INFO] ------------------------------------------------------------------------
>>>>> >>>> >> edward@test ~/workspace/hama-trunk $ ls lib/
>>>>> >>>> >> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>>>> >>>> >> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>>>> >>>> >> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>>>> >>>> >>    zookeeper-3.3.1.jar
>>>>> >>>> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
>>>>> >>>> >> -bash: bin/hama: Permission denied
>>>>> >>>> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
>>>>> >>>> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
>>>>> >>>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>> >>>> >> org/apache/commons/cli/ParseException
>>>>> >>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>>> >>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>> >>>> >>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>>> >>>> >> Caused by: java.lang.ClassNotFoundException:
>>>>> >>>> >> org.apache.commons.cli.ParseException
>>>>> >>>> >>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>>> >>>> >>        at java.security.AccessController.doPrivileged(Native Method)
>>>>> >>>> >>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>>> >>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
>>>>> >>>> >>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>> >>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>>> >>>> >>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
>>>>> >>>> >>        ... 3 more
>>>>> >>>> >>
>>>>> >>>> >>
>>>>> >>>> >>
>>>>> >>>> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>> >>>> >>> That's so weird.
>>>>> >>>> >>>
>>>>> >>>> >>> It works well on my test cluster.
>>>>> >>>> >>>
>>>>> >>>> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
>>>>> >>>> >>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>>>> >>>> >>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>>>> >>>> >>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>>>> >>>> >>>    zookeeper-3.3.1.jar
>>>>> >>>> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
>>>>> >>>> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
>>>>> >>>> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
>>>>> >>>> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
>>>>> >>>> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
>>>>> >>>> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
>>>>> >>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
>>>>> >>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
>>>>> >>>> >>> Estimated value of PI is 3.1534
>>>>> >>>> >>> Job Finished in 7.886 seconds
>>>>> >>>> >>> edward@slave:~/workspace/hama-trunk$
>>>>> >>>> >>>
>>>>> >>>> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
>>>>> >>>> >>> <jo...@cbsinteractive.com> wrote:
>>>>> >>>> >>>> Replying to my own post ...
>>>>> >>>> >>>>
>>>>> >>>> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>>>>> >>>> >>>> <jo...@cbsinteractive.com> wrote:
>>>>> >>>> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>> When I try to run the hama daemons using the command -
>>>>> >>>> >>>>>> bin/hama zookeeper
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>> I get this error -
>>>>> >>>> >>>>
>>>>> >>>> >>>> It looks like the problem is actually this line in pom.xml (line 205)  :
>>>>> >>>> >>>>                    <excludeTransitive>true</excludeTransitive>
>>>>> >>>> >>>>
>>>>> >>>> >>>> This prevents commons-cli (and other transitive dependencies) from
>>>>> >>>> >>>> being copied to the lib dir like they should be.
>>>>> >>>> >>>>
>>>>> >>>> >>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
>>>>> >>>> >>>> JIRA for this after lunch, the command line is fairly broken with this
>>>>> >>>> >>>> line in, especially after several dependencies were (correctly)
>>>>> >>>> >>>> removed in HAMA-402.
>>>>> >>>> >>>>
>>>>> >>>> >>>>
>>>>> >>>> >>>> ...joe
>>>>> >>>> >>>>
>>>>> >>>> >>>>
>>>>> >>>> >>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
>>>>> >>>> >>>>> commons-cli dependency was removed recently as part of HAMA-402, and
>>>>> >>>> >>>>> re-adding it fixes the NoClassDefFoundError.
>>>>> >>>> >>>>>
>>>>> >>>> >>>>> I believe commons-cli should have been pulled as part of the
>>>>> >>>> >>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
>>>>> >>>> >>>>> rusty, and I'm no longer sure just how transitive dependencies are
>>>>> >>>> >>>>> pulled)
>>>>> >>>> >>>>>
>>>>> >>>> >>>>> While a quick work-around is to re-add the dependency here, the proper
>>>>> >>>> >>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
>>>>> >>>> >>>>> depends on commons-cli.
>>>>> >>>> >>>>>
>>>>> >>>> >>>>>
>>>>> >>>> >>>>> ...joe
>>>>> >>>> >>>>>
>>>>> >>>> >>>>>
>>>>> >>>> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>> >>>> >>>>>> org/apache/commons/cli/ParseException
>>>>> >>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>>> >>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>> >>>> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>>> >>>> >>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>> >>>> >>>>>> org.apache.commons.cli.ParseException
>>>>> >>>> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>> >>>> >>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> >>>> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>> >>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>> >>>> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>> >>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>>> >>>> >>>>>> ... 3 more
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>> I tried installing it a few times, but can't think of a solution to the
>>>>> >>>> >>>>>> problem. Can you please help ?
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>> Thank you
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>> Sincerely,
>>>>> >>>> >>>>>> Ashish
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>>> Hi,
>>>>> >>>> >>>>>>>
>>>>> >>>> >>>>>>> I would suggest following steps to debug:
>>>>> >>>> >>>>>>>
>>>>> >>>> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>>>>> >>>> >>>>>>> 2. Run the Hama deamons with 3 terminals.
>>>>> >>>> >>>>>>>
>>>>> >>>> >>>>>>> % bin/hama zookeeper
>>>>> >>>> >>>>>>> % bin/hama bspmaster
>>>>> >>>> >>>>>>> % bin/hama groom
>>>>> >>>> >>>>>>>
>>>>> >>>> >>>>>>> 3. Use the logging API to log debug.
>>>>> >>>> >>>>>>> For example, LOG.debug("like this");
>>>>> >>>> >>>>>>>
>>>>> >>>> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>>>>> >>>> >>>>>>> wrote:
>>>>> >>>> >>>>>>> > Hi,
>>>>> >>>> >>>>>>> >
>>>>> >>>> >>>>>>> > I was wondering if anyone can give me instructions on how to debug code
>>>>> >>>> >>>>>>> in
>>>>> >>>> >>>>>>> > distributed or pseudo-distributed mode ? It would really help me.
>>>>> >>>> >>>>>>> >
>>>>> >>>> >>>>>>> > Thank you
>>>>> >>>> >>>>>>> >
>>>>> >>>> >>>>>>> > Sincerely,
>>>>> >>>> >>>>>>> > Ashish
>>>>> >>>> >>>>>>> >
>>>>> >>>> >>>>>>> >
>>>>> >>>> >>>>>>> >
>>>>> >>>> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>>>>> >>>> >>>>>>> >wrote:
>>>>> >>>> >>>>>>> >
>>>>> >>>> >>>>>>> >>
>>>>> >>>> >>>>>>> >> Hi,
>>>>> >>>> >>>>>>> >>
>>>>> >>>> >>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>>>>> >>>> >>>>>>> to
>>>>> >>>> >>>>>>> >> debug.
>>>>> >>>> >>>>>>> >>
>>>>> >>>> >>>>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
>>>>> >>>> >>>>>>> can
>>>>> >>>> >>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>>>>> >>>> >>>>>>> be
>>>>> >>>> >>>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>>>>> >>>> >>>>>>> >> server), but where do I specify that in the code in eclipse ?
>>>>> >>>> >>>>>>> >>
>>>>> >>>> >>>>>>> >> My next question is - what is the difference between MiniBSPCluster and
>>>>> >>>> >>>>>>> >> LocalBSPRunner  and which file should I be using to be able to step
>>>>> >>>> >>>>>>> through
>>>>> >>>> >>>>>>> >> the code ?
>>>>> >>>> >>>>>>> >>
>>>>> >>>> >>>>>>> >> These are the processes currently running on my machine -
>>>>> >>>> >>>>>>> >>
>>>>> >>>> >>>>>>> >> 6112 NameNode
>>>>> >>>> >>>>>>> >> 3319 ZooKeeperRunner
>>>>> >>>> >>>>>>> >> 3018
>>>>> >>>> >>>>>>> >> 6449 SecondaryNameNode
>>>>> >>>> >>>>>>> >> 5709 HMaster
>>>>> >>>> >>>>>>> >> 5643 HQuorumPeer
>>>>> >>>> >>>>>>> >> 6684 TaskTracker
>>>>> >>>> >>>>>>> >> 6526 JobTracker
>>>>> >>>> >>>>>>> >> 8074 Jps
>>>>> >>>> >>>>>>> >> 6277 DataNode
>>>>> >>>> >>>>>>> >>
>>>>> >>>> >>>>>>> >> Thank you,
>>>>> >>>> >>>>>>> >> Ashish
>>>>> >>>> >>>>>>> >>
>>>>> >>>> >>>>>>> >
>>>>> >>>> >>>>>>>
>>>>> >>>> >>>>>>>
>>>>> >>>> >>>>>>>
>>>>> >>>> >>>>>>> --
>>>>> >>>> >>>>>>> Best Regards, Edward J. Yoon
>>>>> >>>> >>>>>>> @eddieyoon
>>>>> >>>> >>>>>>>
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>
>>>>> >>>> >>>>
>>>>> >>>> >>>
>>>>> >>>> >>>
>>>>> >>>> >>>
>>>>> >>>> >>> --
>>>>> >>>> >>> Best Regards, Edward J. Yoon
>>>>> >>>> >>> @eddieyoon
>>>>> >>>> >>>
>>>>> >>>> >>
>>>>> >>>> >>
>>>>> >>>> >>
>>>>> >>>> >> --
>>>>> >>>> >> Best Regards, Edward J. Yoon
>>>>> >>>> >> @eddieyoon
>>>>> >>>> >>
>>>>> >>>> >
>>>>> >>>> >
>>>>> >>>> >
>>>>> >>>> > --
>>>>> >>>> > Best Regards, Edward J. Yoon
>>>>> >>>> > @eddieyoon
>>>>> >>>> >
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards, Edward J. Yoon
>>>>> >> @eddieyoon
>>>>> >>
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards, Edward J. Yoon
>>>>> @eddieyoon
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>>
>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>

Re: Debugging hama in eclipse

Posted by "Edward J. Yoon" <ed...@apache.org>.

Have you tried to run with <value>ubuntu</value>?

On Thu, Jun 23, 2011 at 6:04 PM, Ashish Agarwal <as...@gmail.com> wrote:
> <property>
>   <name>fs.default.name</name>
>    <value>localhost:9000</value>
>  </property>
>
> This is the same in both hadoop-site.xml and the hama-site.xml
>
> The /etc/hosts file looks like this -
>
> 69.90.54.2      ubuntu  # Added by NetworkManager
> 127.0.0.1       localhost.localdomain   localhost
> ::1     ubuntu  localhost6.localdomain6 localhost6
> 127.0.1.1       ubuntu.ubuntu-domain    ubuntu
>
> # The following lines are desirable for IPv6 capable hosts
> ::1     localhost ip6-localhost ip6-loopback
> fe00::0 ip6-localnet
> ff00::0 ip6-mcastprefix
> ff02::1 ip6-allnodes
> ff02::2 ip6-allrouters
> ff02::3 ip6-allhosts
>
> Thanks
> Ashish
>
>
> On Thu, Jun 23, 2011 at 1:40 AM, Edward J. Yoon <ed...@apache.org> wrote:
>> What is your 'fs.default.name' of hadoop?
>>
>> What is your hostname?
>>
>> Have you seen the /etc/hosts file?
>>
>> On Thu, Jun 23, 2011 at 5:22 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>> I already have made that change, but I still have the same error
>>>
>>> I have tried
>>> <value>localhost:40000</value>
>>> and also
>>> <value>local</value>
>>>
>>>
>>> Thanks
>>> Ashish
>>>
>>>
>>>
>>> On Thu, Jun 23, 2011 at 1:12 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>>
>>>> You need to set the 'bsp.master.address' to localhost or your hostname.
>>>>
>>>> conf/hama-site.xml:
>>>>
>>>>  <property>
>>>>    <name>bsp.master.address</name>
>>>>    <value>localhost:40000</value>
>>>>  </property>
>>>>
>>>> On Thu, Jun 23, 2011 at 5:01 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>>> > Now, I am having a problem starting the bspmaster :(
>>>> > bin/hama bspmaster
>>>> >
>>>> >
>>>> > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama bspmaster
>>>> > 11/06/23 01:00:13 INFO hama.BSPMasterRunner: STARTUP_MSG:
>>>> > /************************************************************
>>>> > STARTUP_MSG: Starting BSPMaster
>>>> > STARTUP_MSG:   host = ubuntu/127.0.1.1
>>>> > STARTUP_MSG:   args = []
>>>> > STARTUP_MSG:   version = 0.20.2
>>>> > STARTUP_MSG:   build =  -r ; compiled by 'chrisdo' on Wed Mar 10
>>>> > 13:22:54 PST 2010
>>>> > ************************************************************/
>>>> > 11/06/23 01:00:13 INFO bsp.BSPMaster: RPC BSPMaster: host local port 40000
>>>> > 11/06/23 01:00:14 FATAL hama.BSPMasterRunner:
>>>> > java.net.UnknownHostException: Invalid hostname for server: local
>>>> >        at org.apache.hadoop.ipc.Server.bind(Server.java:198)
>>>> >        at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
>>>> >        at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
>>>> >        at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
>>>> >        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
>>>> >        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:441)
>>>> >        at org.apache.hama.bsp.BSPMaster.<init>(BSPMaster.java:248)
>>>> >        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:419)
>>>> >        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:413)
>>>> >        at org.apache.hama.BSPMasterRunner.run(BSPMasterRunner.java:46)
>>>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>> >        at org.apache.hama.BSPMasterRunner.main(BSPMasterRunner.java:56)
>>>> >
>>>> > 11/06/23 01:00:14 INFO hama.BSPMasterRunner: SHUTDOWN_MSG:
>>>> > /************************************************************
>>>> > SHUTDOWN_MSG: Shutting down BSPMaster at ubuntu/127.0.1.1
>>>> > ************************************************************/
>>>> > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>>>> >
>>>> >
>>>> > Thanks
>>>> > Ashish
>>>> >
>>>> > On Thu, Jun 23, 2011 at 12:49 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>> >> If you want to execute a command in the background without watching
>>>> >> logs, you can do the following:
>>>> >>
>>>> >> % bin/hama zookeeper &
>>>> >>
>>>> >> On Thu, Jun 23, 2011 at 4:43 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>>> >>> How long does it take to execute the command
>>>> >>> bin/hama zookeeper
>>>> >>>
>>>> >>> I had to stop the execution as it stopped doing anything after some time.
>>>> >>>
>>>> >>> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>> >>> environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
>>>> >>> GMT
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>> >>> environment:host.name=ubuntu.ubuntu-domain
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>> >>> environment:java.version=1.6.0_24
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>> >>> environment:java.vendor=Sun Microsystems Inc.
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>> >>> environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>> >>> environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>> >>> environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>> >>> environment:java.io.tmpdir=/tmp
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>> >>> environment:java.compiler=<NA>
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.name=Linux
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.arch=i386
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>> >>> environment:os.version=2.6.35-28-generic
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>> >>> environment:user.name=hadoop
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>> >>> environment:user.home=/home/hadoop
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>>> >>> environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1
>>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to 1200000
>>>> >>> 11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port
>>>> >>> 0.0.0.0/0.0.0.0:21810
>>>> >>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>>>> >>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>>>> >>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>>>> >>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>>>> >>> 11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0
>>>> >>> ^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>>>> >>>
>>>> >>>
>>>> >>> Thanks
>>>> >>> Ashish
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd
>>>> >>> <jo...@cbsinteractive.com> wrote:
>>>> >>>>
>>>> >>>> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <ed...@apache.org> wrote:
>>>> >>>> > Instead removing 206 line, why don't we add dependencies as below?
>>>> >>>>
>>>> >>>> I think removing line 206 is the right thing to do.  Hama doesn't
>>>> >>>> depend on commons-cli, hadoop-core does, and maven will properly pick
>>>> >>>> that up and include the jar if allowed.
>>>> >>>>
>>>> >>>> The fact that Hama doesn't depend on the several items below was
>>>> >>>> correctly detected by maven, in the report description of HAMA-402.
>>>> >>>>
>>>> >>>> You don't mention a particular problem with including the transitive
>>>> >>>> dependencies, but if your problem is that it includes too many items,
>>>> >>>> maven does have options to exclude a particular transitive dependency
>>>> >>>> on a case-by-case basis that might be more appropriate to use.
>>>> >>>>
>>>> >>>> Including the dependencies below as you suggest puts Hama in the place
>>>> >>>> of maintaining a transitive dependency like commons-cli in lock-step
>>>> >>>> hadoop-core, where the actual dependency comes from.
>>>> >>>>
>>>> >>>>
>>>> >>>> This is all just my $.02 worth, I'm open to arguments in favor of
>>>> >>>> <excludeTransitive/>, and there is often more than one way to skin a
>>>> >>>> cat with respect to dependency management.
>>>> >>>>
>>>> >>>>
>>>> >>>> ...joe
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> >    <dependency>
>>>> >>>> >      <groupId>org.mortbay.jetty</groupId>
>>>> >>>> >      <artifactId>jetty-annotations</artifactId>
>>>> >>>> >      <version>6.1H.22</version>
>>>> >>>> >    </dependency>
>>>> >>>> >    <dependency>
>>>> >>>> >      <groupId>org.slf4j</groupId>
>>>> >>>> >      <artifactId>slf4j-log4j12</artifactId>
>>>> >>>> >      <version>1.5.2</version>
>>>> >>>> >    </dependency>
>>>> >>>> >    <dependency>
>>>> >>>> >      <groupId>commons-cli</groupId>
>>>> >>>> >      <artifactId>commons-cli</artifactId>
>>>> >>>> >      <version>1.2</version>
>>>> >>>> >    </dependency>
>>>> >>>> >
>>>> >>>> > Thanks.
>>>> >>>> >
>>>> >>>> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>> >>>> >> Oh... sorry, you're right.
>>>> >>>> >>
>>>> >>>> >> It looks like a bug. We have to fix this.
>>>> >>>> >>
>>>> >>>> >> [INFO] Total time: 4:36.098s
>>>> >>>> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
>>>> >>>> >> [INFO] Final Memory: 20M/51M
>>>> >>>> >> [INFO] ------------------------------------------------------------------------
>>>> >>>> >> edward@test ~/workspace/hama-trunk $ ls lib/
>>>> >>>> >> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>>> >>>> >> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>>> >>>> >> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>>> >>>> >>    zookeeper-3.3.1.jar
>>>> >>>> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
>>>> >>>> >> -bash: bin/hama: Permission denied
>>>> >>>> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
>>>> >>>> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
>>>> >>>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> >>>> >> org/apache/commons/cli/ParseException
>>>> >>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>> >>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>> >>>> >>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>> >>>> >> Caused by: java.lang.ClassNotFoundException:
>>>> >>>> >> org.apache.commons.cli.ParseException
>>>> >>>> >>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>> >>>> >>        at java.security.AccessController.doPrivileged(Native Method)
>>>> >>>> >>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>> >>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
>>>> >>>> >>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> >>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>> >>>> >>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
>>>> >>>> >>        ... 3 more
>>>> >>>> >>
>>>> >>>> >>
>>>> >>>> >>
>>>> >>>> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>> >>>> >>> That's so weird.
>>>> >>>> >>>
>>>> >>>> >>> It works well on my test cluster.
>>>> >>>> >>>
>>>> >>>> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
>>>> >>>> >>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>>> >>>> >>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>>> >>>> >>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>>> >>>> >>>    zookeeper-3.3.1.jar
>>>> >>>> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
>>>> >>>> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
>>>> >>>> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
>>>> >>>> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
>>>> >>>> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
>>>> >>>> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
>>>> >>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
>>>> >>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
>>>> >>>> >>> Estimated value of PI is 3.1534
>>>> >>>> >>> Job Finished in 7.886 seconds
>>>> >>>> >>> edward@slave:~/workspace/hama-trunk$
>>>> >>>> >>>
>>>> >>>> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
>>>> >>>> >>> <jo...@cbsinteractive.com> wrote:
>>>> >>>> >>>> Replying to my own post ...
>>>> >>>> >>>>
>>>> >>>> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>>>> >>>> >>>> <jo...@cbsinteractive.com> wrote:
>>>> >>>> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> When I try to run the hama daemons using the command -
>>>> >>>> >>>>>> bin/hama zookeeper
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> I get this error -
>>>> >>>> >>>>
>>>> >>>> >>>> It looks like the problem is actually this line in pom.xml (line 205)  :
>>>> >>>> >>>>                    <excludeTransitive>true</excludeTransitive>
>>>> >>>> >>>>
>>>> >>>> >>>> This prevents commons-cli (and other transitive dependencies) from
>>>> >>>> >>>> being copied to the lib dir like they should be.
>>>> >>>> >>>>
>>>> >>>> >>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
>>>> >>>> >>>> JIRA for this after lunch, the command line is fairly broken with this
>>>> >>>> >>>> line in, especially after several dependencies were (correctly)
>>>> >>>> >>>> removed in HAMA-402.
>>>> >>>> >>>>
>>>> >>>> >>>>
>>>> >>>> >>>> ...joe
>>>> >>>> >>>>
>>>> >>>> >>>>
>>>> >>>> >>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
>>>> >>>> >>>>> commons-cli dependency was removed recently as part of HAMA-402, and
>>>> >>>> >>>>> re-adding it fixes the NoClassDefFoundError.
>>>> >>>> >>>>>
>>>> >>>> >>>>> I believe commons-cli should have been pulled as part of the
>>>> >>>> >>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
>>>> >>>> >>>>> rusty, and I'm no longer sure just how transitive dependencies are
>>>> >>>> >>>>> pulled)
>>>> >>>> >>>>>
>>>> >>>> >>>>> While a quick work-around is to re-add the dependency here, the proper
>>>> >>>> >>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
>>>> >>>> >>>>> depends on commons-cli.
>>>> >>>> >>>>>
>>>> >>>> >>>>>
>>>> >>>> >>>>> ...joe
>>>> >>>> >>>>>
>>>> >>>> >>>>>
>>>> >>>> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> >>>> >>>>>> org/apache/commons/cli/ParseException
>>>> >>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>> >>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>> >>>> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>> >>>> >>>>>> Caused by: java.lang.ClassNotFoundException:
>>>> >>>> >>>>>> org.apache.commons.cli.ParseException
>>>> >>>> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> >>>> >>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> >>>> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> >>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>> >>>> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> >>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>> >>>> >>>>>> ... 3 more
>>>> >>>> >>>>>>
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> I tried installing it a few times, but can't think of a solution to the
>>>> >>>> >>>>>> problem. Can you please help ?
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> Thank you
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> Sincerely,
>>>> >>>> >>>>>> Ashish
>>>> >>>> >>>>>>
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>>>> >>>> >>>>>>
>>>> >>>> >>>>>>> Hi,
>>>> >>>> >>>>>>>
>>>> >>>> >>>>>>> I would suggest following steps to debug:
>>>> >>>> >>>>>>>
>>>> >>>> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>>>> >>>> >>>>>>> 2. Run the Hama deamons with 3 terminals.
>>>> >>>> >>>>>>>
>>>> >>>> >>>>>>> % bin/hama zookeeper
>>>> >>>> >>>>>>> % bin/hama bspmaster
>>>> >>>> >>>>>>> % bin/hama groom
>>>> >>>> >>>>>>>
>>>> >>>> >>>>>>> 3. Use the logging API to log debug.
>>>> >>>> >>>>>>> For example, LOG.debug("like this");
>>>> >>>> >>>>>>>
>>>> >>>> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>>>> >>>> >>>>>>> wrote:
>>>> >>>> >>>>>>> > Hi,
>>>> >>>> >>>>>>> >
>>>> >>>> >>>>>>> > I was wondering if anyone can give me instructions on how to debug code
>>>> >>>> >>>>>>> in
>>>> >>>> >>>>>>> > distributed or pseudo-distributed mode ? It would really help me.
>>>> >>>> >>>>>>> >
>>>> >>>> >>>>>>> > Thank you
>>>> >>>> >>>>>>> >
>>>> >>>> >>>>>>> > Sincerely,
>>>> >>>> >>>>>>> > Ashish
>>>> >>>> >>>>>>> >
>>>> >>>> >>>>>>> >
>>>> >>>> >>>>>>> >
>>>> >>>> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>>>> >>>> >>>>>>> >wrote:
>>>> >>>> >>>>>>> >
>>>> >>>> >>>>>>> >>
>>>> >>>> >>>>>>> >> Hi,
>>>> >>>> >>>>>>> >>
>>>> >>>> >>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>>>> >>>> >>>>>>> to
>>>> >>>> >>>>>>> >> debug.
>>>> >>>> >>>>>>> >>
>>>> >>>> >>>>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
>>>> >>>> >>>>>>> can
>>>> >>>> >>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>>>> >>>> >>>>>>> be
>>>> >>>> >>>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>>>> >>>> >>>>>>> >> server), but where do I specify that in the code in eclipse ?
>>>> >>>> >>>>>>> >>
>>>> >>>> >>>>>>> >> My next question is - what is the difference between MiniBSPCluster and
>>>> >>>> >>>>>>> >> LocalBSPRunner  and which file should I be using to be able to step
>>>> >>>> >>>>>>> through
>>>> >>>> >>>>>>> >> the code ?
>>>> >>>> >>>>>>> >>
>>>> >>>> >>>>>>> >> These are the processes currently running on my machine -
>>>> >>>> >>>>>>> >>
>>>> >>>> >>>>>>> >> 6112 NameNode
>>>> >>>> >>>>>>> >> 3319 ZooKeeperRunner
>>>> >>>> >>>>>>> >> 3018
>>>> >>>> >>>>>>> >> 6449 SecondaryNameNode
>>>> >>>> >>>>>>> >> 5709 HMaster
>>>> >>>> >>>>>>> >> 5643 HQuorumPeer
>>>> >>>> >>>>>>> >> 6684 TaskTracker
>>>> >>>> >>>>>>> >> 6526 JobTracker
>>>> >>>> >>>>>>> >> 8074 Jps
>>>> >>>> >>>>>>> >> 6277 DataNode
>>>> >>>> >>>>>>> >>
>>>> >>>> >>>>>>> >> Thank you,
>>>> >>>> >>>>>>> >> Ashish
>>>> >>>> >>>>>>> >>
>>>> >>>> >>>>>>> >
>>>> >>>> >>>>>>>
>>>> >>>> >>>>>>>
>>>> >>>> >>>>>>>
>>>> >>>> >>>>>>> --
>>>> >>>> >>>>>>> Best Regards, Edward J. Yoon
>>>> >>>> >>>>>>> @eddieyoon
>>>> >>>> >>>>>>>
>>>> >>>> >>>>>>
>>>> >>>> >>>>>
>>>> >>>> >>>>
>>>> >>>> >>>
>>>> >>>> >>>
>>>> >>>> >>>
>>>> >>>> >>> --
>>>> >>>> >>> Best Regards, Edward J. Yoon
>>>> >>>> >>> @eddieyoon
>>>> >>>> >>>
>>>> >>>> >>
>>>> >>>> >>
>>>> >>>> >>
>>>> >>>> >> --
>>>> >>>> >> Best Regards, Edward J. Yoon
>>>> >>>> >> @eddieyoon
>>>> >>>> >>
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> > --
>>>> >>>> > Best Regards, Edward J. Yoon
>>>> >>>> > @eddieyoon
>>>> >>>> >
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Best Regards, Edward J. Yoon
>>>> >> @eddieyoon
>>>> >>
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards, Edward J. Yoon
>>>> @eddieyoon
>>>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: Debugging hama in eclipse

Posted by Ashish Agarwal <as...@gmail.com>.

<property>
   <name>fs.default.name</name>
    <value>localhost:9000</value>
  </property>

This is the same in both hadoop-site.xml and the hama-site.xml

The /etc/hosts file looks like this -

69.90.54.2      ubuntu  # Added by NetworkManager
127.0.0.1       localhost.localdomain   localhost
::1     ubuntu  localhost6.localdomain6 localhost6
127.0.1.1       ubuntu.ubuntu-domain    ubuntu

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Thanks
Ashish


On Thu, Jun 23, 2011 at 1:40 AM, Edward J. Yoon <ed...@apache.org> wrote:
> What is your 'fs.default.name' of hadoop?
>
> What is your hostname?
>
> Have you seen the /etc/hosts file?
>
> On Thu, Jun 23, 2011 at 5:22 PM, Ashish Agarwal <as...@gmail.com> wrote:
>> I already have made that change, but I still have the same error
>>
>> I have tried
>> <value>localhost:40000</value>
>> and also
>> <value>local</value>
>>
>>
>> Thanks
>> Ashish
>>
>>
>>
>> On Thu, Jun 23, 2011 at 1:12 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>
>>> You need to set the 'bsp.master.address' to localhost or your hostname.
>>>
>>> conf/hama-site.xml:
>>>
>>>  <property>
>>>    <name>bsp.master.address</name>
>>>    <value>localhost:40000</value>
>>>  </property>
>>>
>>> On Thu, Jun 23, 2011 at 5:01 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>> > Now, I am having a problem starting the bspmaster :(
>>> > bin/hama bspmaster
>>> >
>>> >
>>> > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama bspmaster
>>> > 11/06/23 01:00:13 INFO hama.BSPMasterRunner: STARTUP_MSG:
>>> > /************************************************************
>>> > STARTUP_MSG: Starting BSPMaster
>>> > STARTUP_MSG:   host = ubuntu/127.0.1.1
>>> > STARTUP_MSG:   args = []
>>> > STARTUP_MSG:   version = 0.20.2
>>> > STARTUP_MSG:   build =  -r ; compiled by 'chrisdo' on Wed Mar 10
>>> > 13:22:54 PST 2010
>>> > ************************************************************/
>>> > 11/06/23 01:00:13 INFO bsp.BSPMaster: RPC BSPMaster: host local port 40000
>>> > 11/06/23 01:00:14 FATAL hama.BSPMasterRunner:
>>> > java.net.UnknownHostException: Invalid hostname for server: local
>>> >        at org.apache.hadoop.ipc.Server.bind(Server.java:198)
>>> >        at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
>>> >        at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
>>> >        at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
>>> >        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
>>> >        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:441)
>>> >        at org.apache.hama.bsp.BSPMaster.<init>(BSPMaster.java:248)
>>> >        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:419)
>>> >        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:413)
>>> >        at org.apache.hama.BSPMasterRunner.run(BSPMasterRunner.java:46)
>>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>> >        at org.apache.hama.BSPMasterRunner.main(BSPMasterRunner.java:56)
>>> >
>>> > 11/06/23 01:00:14 INFO hama.BSPMasterRunner: SHUTDOWN_MSG:
>>> > /************************************************************
>>> > SHUTDOWN_MSG: Shutting down BSPMaster at ubuntu/127.0.1.1
>>> > ************************************************************/
>>> > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>>> >
>>> >
>>> > Thanks
>>> > Ashish
>>> >
>>> > On Thu, Jun 23, 2011 at 12:49 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>> >> If you want to execute a command in the background without watching
>>> >> logs, you can do the following:
>>> >>
>>> >> % bin/hama zookeeper &
>>> >>
>>> >> On Thu, Jun 23, 2011 at 4:43 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>> >>> How long does it take to execute the command
>>> >>> bin/hama zookeeper
>>> >>>
>>> >>> I had to stop the execution as it stopped doing anything after some time.
>>> >>>
>>> >>> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> >>> environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
>>> >>> GMT
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> >>> environment:host.name=ubuntu.ubuntu-domain
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> >>> environment:java.version=1.6.0_24
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> >>> environment:java.vendor=Sun Microsystems Inc.
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> >>> environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> >>> environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> >>> environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> >>> environment:java.io.tmpdir=/tmp
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> >>> environment:java.compiler=<NA>
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.name=Linux
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.arch=i386
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> >>> environment:os.version=2.6.35-28-generic
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> >>> environment:user.name=hadoop
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> >>> environment:user.home=/home/hadoop
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> >>> environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1
>>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to 1200000
>>> >>> 11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port
>>> >>> 0.0.0.0/0.0.0.0:21810
>>> >>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>>> >>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>>> >>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>>> >>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>>> >>> 11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0
>>> >>> ^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>>> >>>
>>> >>>
>>> >>> Thanks
>>> >>> Ashish
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd
>>> >>> <jo...@cbsinteractive.com> wrote:
>>> >>>>
>>> >>>> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <ed...@apache.org> wrote:
>>> >>>> > Instead removing 206 line, why don't we add dependencies as below?
>>> >>>>
>>> >>>> I think removing line 206 is the right thing to do.  Hama doesn't
>>> >>>> depend on commons-cli, hadoop-core does, and maven will properly pick
>>> >>>> that up and include the jar if allowed.
>>> >>>>
>>> >>>> The fact that Hama doesn't depend on the several items below was
>>> >>>> correctly detected by maven, in the report description of HAMA-402.
>>> >>>>
>>> >>>> You don't mention a particular problem with including the transitive
>>> >>>> dependencies, but if your problem is that it includes too many items,
>>> >>>> maven does have options to exclude a particular transitive dependency
>>> >>>> on a case-by-case basis that might be more appropriate to use.
>>> >>>>
>>> >>>> Including the dependencies below as you suggest puts Hama in the place
>>> >>>> of maintaining a transitive dependency like commons-cli in lock-step
>>> >>>> hadoop-core, where the actual dependency comes from.
>>> >>>>
>>> >>>>
>>> >>>> This is all just my $.02 worth, I'm open to arguments in favor of
>>> >>>> <excludeTransitive/>, and there is often more than one way to skin a
>>> >>>> cat with respect to dependency management.
>>> >>>>
>>> >>>>
>>> >>>> ...joe
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> >    <dependency>
>>> >>>> >      <groupId>org.mortbay.jetty</groupId>
>>> >>>> >      <artifactId>jetty-annotations</artifactId>
>>> >>>> >      <version>6.1H.22</version>
>>> >>>> >    </dependency>
>>> >>>> >    <dependency>
>>> >>>> >      <groupId>org.slf4j</groupId>
>>> >>>> >      <artifactId>slf4j-log4j12</artifactId>
>>> >>>> >      <version>1.5.2</version>
>>> >>>> >    </dependency>
>>> >>>> >    <dependency>
>>> >>>> >      <groupId>commons-cli</groupId>
>>> >>>> >      <artifactId>commons-cli</artifactId>
>>> >>>> >      <version>1.2</version>
>>> >>>> >    </dependency>
>>> >>>> >
>>> >>>> > Thanks.
>>> >>>> >
>>> >>>> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>> >>>> >> Oh... sorry, you're right.
>>> >>>> >>
>>> >>>> >> It looks like a bug. We have to fix this.
>>> >>>> >>
>>> >>>> >> [INFO] Total time: 4:36.098s
>>> >>>> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
>>> >>>> >> [INFO] Final Memory: 20M/51M
>>> >>>> >> [INFO] ------------------------------------------------------------------------
>>> >>>> >> edward@test ~/workspace/hama-trunk $ ls lib/
>>> >>>> >> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>> >>>> >> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>> >>>> >> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>> >>>> >>    zookeeper-3.3.1.jar
>>> >>>> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
>>> >>>> >> -bash: bin/hama: Permission denied
>>> >>>> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
>>> >>>> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
>>> >>>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> >>>> >> org/apache/commons/cli/ParseException
>>> >>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>> >>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>> >>>> >>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>> >>>> >> Caused by: java.lang.ClassNotFoundException:
>>> >>>> >> org.apache.commons.cli.ParseException
>>> >>>> >>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>> >>>> >>        at java.security.AccessController.doPrivileged(Native Method)
>>> >>>> >>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>> >>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
>>> >>>> >>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> >>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> >>>> >>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
>>> >>>> >>        ... 3 more
>>> >>>> >>
>>> >>>> >>
>>> >>>> >>
>>> >>>> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>> >>>> >>> That's so weird.
>>> >>>> >>>
>>> >>>> >>> It works well on my test cluster.
>>> >>>> >>>
>>> >>>> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
>>> >>>> >>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>> >>>> >>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>> >>>> >>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>> >>>> >>>    zookeeper-3.3.1.jar
>>> >>>> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
>>> >>>> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
>>> >>>> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
>>> >>>> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
>>> >>>> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
>>> >>>> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
>>> >>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
>>> >>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
>>> >>>> >>> Estimated value of PI is 3.1534
>>> >>>> >>> Job Finished in 7.886 seconds
>>> >>>> >>> edward@slave:~/workspace/hama-trunk$
>>> >>>> >>>
>>> >>>> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
>>> >>>> >>> <jo...@cbsinteractive.com> wrote:
>>> >>>> >>>> Replying to my own post ...
>>> >>>> >>>>
>>> >>>> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>>> >>>> >>>> <jo...@cbsinteractive.com> wrote:
>>> >>>> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>>> >>>> >>>>>>
>>> >>>> >>>>>> When I try to run the hama daemons using the command -
>>> >>>> >>>>>> bin/hama zookeeper
>>> >>>> >>>>>>
>>> >>>> >>>>>> I get this error -
>>> >>>> >>>>
>>> >>>> >>>> It looks like the problem is actually this line in pom.xml (line 205)  :
>>> >>>> >>>>                    <excludeTransitive>true</excludeTransitive>
>>> >>>> >>>>
>>> >>>> >>>> This prevents commons-cli (and other transitive dependencies) from
>>> >>>> >>>> being copied to the lib dir like they should be.
>>> >>>> >>>>
>>> >>>> >>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
>>> >>>> >>>> JIRA for this after lunch, the command line is fairly broken with this
>>> >>>> >>>> line in, especially after several dependencies were (correctly)
>>> >>>> >>>> removed in HAMA-402.
>>> >>>> >>>>
>>> >>>> >>>>
>>> >>>> >>>> ...joe
>>> >>>> >>>>
>>> >>>> >>>>
>>> >>>> >>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
>>> >>>> >>>>> commons-cli dependency was removed recently as part of HAMA-402, and
>>> >>>> >>>>> re-adding it fixes the NoClassDefFoundError.
>>> >>>> >>>>>
>>> >>>> >>>>> I believe commons-cli should have been pulled as part of the
>>> >>>> >>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
>>> >>>> >>>>> rusty, and I'm no longer sure just how transitive dependencies are
>>> >>>> >>>>> pulled)
>>> >>>> >>>>>
>>> >>>> >>>>> While a quick work-around is to re-add the dependency here, the proper
>>> >>>> >>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
>>> >>>> >>>>> depends on commons-cli.
>>> >>>> >>>>>
>>> >>>> >>>>>
>>> >>>> >>>>> ...joe
>>> >>>> >>>>>
>>> >>>> >>>>>
>>> >>>> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> >>>> >>>>>> org/apache/commons/cli/ParseException
>>> >>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>> >>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>> >>>> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>> >>>> >>>>>> Caused by: java.lang.ClassNotFoundException:
>>> >>>> >>>>>> org.apache.commons.cli.ParseException
>>> >>>> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> >>>> >>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>> >>>> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> >>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> >>>> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> >>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> >>>> >>>>>> ... 3 more
>>> >>>> >>>>>>
>>> >>>> >>>>>>
>>> >>>> >>>>>> I tried installing it a few times, but can't think of a solution to the
>>> >>>> >>>>>> problem. Can you please help ?
>>> >>>> >>>>>>
>>> >>>> >>>>>> Thank you
>>> >>>> >>>>>>
>>> >>>> >>>>>> Sincerely,
>>> >>>> >>>>>> Ashish
>>> >>>> >>>>>>
>>> >>>> >>>>>>
>>> >>>> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>>> >>>> >>>>>>
>>> >>>> >>>>>>> Hi,
>>> >>>> >>>>>>>
>>> >>>> >>>>>>> I would suggest following steps to debug:
>>> >>>> >>>>>>>
>>> >>>> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>>> >>>> >>>>>>> 2. Run the Hama deamons with 3 terminals.
>>> >>>> >>>>>>>
>>> >>>> >>>>>>> % bin/hama zookeeper
>>> >>>> >>>>>>> % bin/hama bspmaster
>>> >>>> >>>>>>> % bin/hama groom
>>> >>>> >>>>>>>
>>> >>>> >>>>>>> 3. Use the logging API to log debug.
>>> >>>> >>>>>>> For example, LOG.debug("like this");
>>> >>>> >>>>>>>
>>> >>>> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>>> >>>> >>>>>>> wrote:
>>> >>>> >>>>>>> > Hi,
>>> >>>> >>>>>>> >
>>> >>>> >>>>>>> > I was wondering if anyone can give me instructions on how to debug code
>>> >>>> >>>>>>> in
>>> >>>> >>>>>>> > distributed or pseudo-distributed mode ? It would really help me.
>>> >>>> >>>>>>> >
>>> >>>> >>>>>>> > Thank you
>>> >>>> >>>>>>> >
>>> >>>> >>>>>>> > Sincerely,
>>> >>>> >>>>>>> > Ashish
>>> >>>> >>>>>>> >
>>> >>>> >>>>>>> >
>>> >>>> >>>>>>> >
>>> >>>> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>>> >>>> >>>>>>> >wrote:
>>> >>>> >>>>>>> >
>>> >>>> >>>>>>> >>
>>> >>>> >>>>>>> >> Hi,
>>> >>>> >>>>>>> >>
>>> >>>> >>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>>> >>>> >>>>>>> to
>>> >>>> >>>>>>> >> debug.
>>> >>>> >>>>>>> >>
>>> >>>> >>>>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
>>> >>>> >>>>>>> can
>>> >>>> >>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>>> >>>> >>>>>>> be
>>> >>>> >>>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>>> >>>> >>>>>>> >> server), but where do I specify that in the code in eclipse ?
>>> >>>> >>>>>>> >>
>>> >>>> >>>>>>> >> My next question is - what is the difference between MiniBSPCluster and
>>> >>>> >>>>>>> >> LocalBSPRunner  and which file should I be using to be able to step
>>> >>>> >>>>>>> through
>>> >>>> >>>>>>> >> the code ?
>>> >>>> >>>>>>> >>
>>> >>>> >>>>>>> >> These are the processes currently running on my machine -
>>> >>>> >>>>>>> >>
>>> >>>> >>>>>>> >> 6112 NameNode
>>> >>>> >>>>>>> >> 3319 ZooKeeperRunner
>>> >>>> >>>>>>> >> 3018
>>> >>>> >>>>>>> >> 6449 SecondaryNameNode
>>> >>>> >>>>>>> >> 5709 HMaster
>>> >>>> >>>>>>> >> 5643 HQuorumPeer
>>> >>>> >>>>>>> >> 6684 TaskTracker
>>> >>>> >>>>>>> >> 6526 JobTracker
>>> >>>> >>>>>>> >> 8074 Jps
>>> >>>> >>>>>>> >> 6277 DataNode
>>> >>>> >>>>>>> >>
>>> >>>> >>>>>>> >> Thank you,
>>> >>>> >>>>>>> >> Ashish
>>> >>>> >>>>>>> >>
>>> >>>> >>>>>>> >
>>> >>>> >>>>>>>
>>> >>>> >>>>>>>
>>> >>>> >>>>>>>
>>> >>>> >>>>>>> --
>>> >>>> >>>>>>> Best Regards, Edward J. Yoon
>>> >>>> >>>>>>> @eddieyoon
>>> >>>> >>>>>>>
>>> >>>> >>>>>>
>>> >>>> >>>>>
>>> >>>> >>>>
>>> >>>> >>>
>>> >>>> >>>
>>> >>>> >>>
>>> >>>> >>> --
>>> >>>> >>> Best Regards, Edward J. Yoon
>>> >>>> >>> @eddieyoon
>>> >>>> >>>
>>> >>>> >>
>>> >>>> >>
>>> >>>> >>
>>> >>>> >> --
>>> >>>> >> Best Regards, Edward J. Yoon
>>> >>>> >> @eddieyoon
>>> >>>> >>
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > --
>>> >>>> > Best Regards, Edward J. Yoon
>>> >>>> > @eddieyoon
>>> >>>> >
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Best Regards, Edward J. Yoon
>>> >> @eddieyoon
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>

Re: Debugging hama in eclipse

Posted by "Edward J. Yoon" <ed...@apache.org>.

What is your 'fs.default.name' of hadoop?

What is your hostname?

Have you seen the /etc/hosts file?

On Thu, Jun 23, 2011 at 5:22 PM, Ashish Agarwal <as...@gmail.com> wrote:
> I already have made that change, but I still have the same error
>
> I have tried
> <value>localhost:40000</value>
> and also
> <value>local</value>
>
>
> Thanks
> Ashish
>
>
>
> On Thu, Jun 23, 2011 at 1:12 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>
>> You need to set the 'bsp.master.address' to localhost or your hostname.
>>
>> conf/hama-site.xml:
>>
>>  <property>
>>    <name>bsp.master.address</name>
>>    <value>localhost:40000</value>
>>  </property>
>>
>> On Thu, Jun 23, 2011 at 5:01 PM, Ashish Agarwal <as...@gmail.com> wrote:
>> > Now, I am having a problem starting the bspmaster :(
>> > bin/hama bspmaster
>> >
>> >
>> > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama bspmaster
>> > 11/06/23 01:00:13 INFO hama.BSPMasterRunner: STARTUP_MSG:
>> > /************************************************************
>> > STARTUP_MSG: Starting BSPMaster
>> > STARTUP_MSG:   host = ubuntu/127.0.1.1
>> > STARTUP_MSG:   args = []
>> > STARTUP_MSG:   version = 0.20.2
>> > STARTUP_MSG:   build =  -r ; compiled by 'chrisdo' on Wed Mar 10
>> > 13:22:54 PST 2010
>> > ************************************************************/
>> > 11/06/23 01:00:13 INFO bsp.BSPMaster: RPC BSPMaster: host local port 40000
>> > 11/06/23 01:00:14 FATAL hama.BSPMasterRunner:
>> > java.net.UnknownHostException: Invalid hostname for server: local
>> >        at org.apache.hadoop.ipc.Server.bind(Server.java:198)
>> >        at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
>> >        at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
>> >        at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
>> >        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
>> >        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:441)
>> >        at org.apache.hama.bsp.BSPMaster.<init>(BSPMaster.java:248)
>> >        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:419)
>> >        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:413)
>> >        at org.apache.hama.BSPMasterRunner.run(BSPMasterRunner.java:46)
>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>> >        at org.apache.hama.BSPMasterRunner.main(BSPMasterRunner.java:56)
>> >
>> > 11/06/23 01:00:14 INFO hama.BSPMasterRunner: SHUTDOWN_MSG:
>> > /************************************************************
>> > SHUTDOWN_MSG: Shutting down BSPMaster at ubuntu/127.0.1.1
>> > ************************************************************/
>> > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>> >
>> >
>> > Thanks
>> > Ashish
>> >
>> > On Thu, Jun 23, 2011 at 12:49 AM, Edward J. Yoon <ed...@apache.org> wrote:
>> >> If you want to execute a command in the background without watching
>> >> logs, you can do the following:
>> >>
>> >> % bin/hama zookeeper &
>> >>
>> >> On Thu, Jun 23, 2011 at 4:43 PM, Ashish Agarwal <as...@gmail.com> wrote:
>> >>> How long does it take to execute the command
>> >>> bin/hama zookeeper
>> >>>
>> >>> I had to stop the execution as it stopped doing anything after some time.
>> >>>
>> >>> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> >>> environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
>> >>> GMT
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> >>> environment:host.name=ubuntu.ubuntu-domain
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> >>> environment:java.version=1.6.0_24
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> >>> environment:java.vendor=Sun Microsystems Inc.
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> >>> environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> >>> environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> >>> environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> >>> environment:java.io.tmpdir=/tmp
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> >>> environment:java.compiler=<NA>
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.name=Linux
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.arch=i386
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> >>> environment:os.version=2.6.35-28-generic
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> >>> environment:user.name=hadoop
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> >>> environment:user.home=/home/hadoop
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> >>> environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1
>> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to 1200000
>> >>> 11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port
>> >>> 0.0.0.0/0.0.0.0:21810
>> >>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>> >>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>> >>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>> >>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>> >>> 11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0
>> >>> ^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>> >>>
>> >>>
>> >>> Thanks
>> >>> Ashish
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd
>> >>> <jo...@cbsinteractive.com> wrote:
>> >>>>
>> >>>> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <ed...@apache.org> wrote:
>> >>>> > Instead removing 206 line, why don't we add dependencies as below?
>> >>>>
>> >>>> I think removing line 206 is the right thing to do.  Hama doesn't
>> >>>> depend on commons-cli, hadoop-core does, and maven will properly pick
>> >>>> that up and include the jar if allowed.
>> >>>>
>> >>>> The fact that Hama doesn't depend on the several items below was
>> >>>> correctly detected by maven, in the report description of HAMA-402.
>> >>>>
>> >>>> You don't mention a particular problem with including the transitive
>> >>>> dependencies, but if your problem is that it includes too many items,
>> >>>> maven does have options to exclude a particular transitive dependency
>> >>>> on a case-by-case basis that might be more appropriate to use.
>> >>>>
>> >>>> Including the dependencies below as you suggest puts Hama in the place
>> >>>> of maintaining a transitive dependency like commons-cli in lock-step
>> >>>> hadoop-core, where the actual dependency comes from.
>> >>>>
>> >>>>
>> >>>> This is all just my $.02 worth, I'm open to arguments in favor of
>> >>>> <excludeTransitive/>, and there is often more than one way to skin a
>> >>>> cat with respect to dependency management.
>> >>>>
>> >>>>
>> >>>> ...joe
>> >>>>
>> >>>>
>> >>>>
>> >>>> >    <dependency>
>> >>>> >      <groupId>org.mortbay.jetty</groupId>
>> >>>> >      <artifactId>jetty-annotations</artifactId>
>> >>>> >      <version>6.1H.22</version>
>> >>>> >    </dependency>
>> >>>> >    <dependency>
>> >>>> >      <groupId>org.slf4j</groupId>
>> >>>> >      <artifactId>slf4j-log4j12</artifactId>
>> >>>> >      <version>1.5.2</version>
>> >>>> >    </dependency>
>> >>>> >    <dependency>
>> >>>> >      <groupId>commons-cli</groupId>
>> >>>> >      <artifactId>commons-cli</artifactId>
>> >>>> >      <version>1.2</version>
>> >>>> >    </dependency>
>> >>>> >
>> >>>> > Thanks.
>> >>>> >
>> >>>> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <ed...@apache.org> wrote:
>> >>>> >> Oh... sorry, you're right.
>> >>>> >>
>> >>>> >> It looks like a bug. We have to fix this.
>> >>>> >>
>> >>>> >> [INFO] Total time: 4:36.098s
>> >>>> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
>> >>>> >> [INFO] Final Memory: 20M/51M
>> >>>> >> [INFO] ------------------------------------------------------------------------
>> >>>> >> edward@test ~/workspace/hama-trunk $ ls lib/
>> >>>> >> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>> >>>> >> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>> >>>> >> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>> >>>> >>    zookeeper-3.3.1.jar
>> >>>> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
>> >>>> >> -bash: bin/hama: Permission denied
>> >>>> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
>> >>>> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
>> >>>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>> >>>> >> org/apache/commons/cli/ParseException
>> >>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>> >>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>> >>>> >>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>> >>>> >> Caused by: java.lang.ClassNotFoundException:
>> >>>> >> org.apache.commons.cli.ParseException
>> >>>> >>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> >>>> >>        at java.security.AccessController.doPrivileged(Native Method)
>> >>>> >>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> >>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
>> >>>> >>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> >>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> >>>> >>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
>> >>>> >>        ... 3 more
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <ed...@apache.org> wrote:
>> >>>> >>> That's so weird.
>> >>>> >>>
>> >>>> >>> It works well on my test cluster.
>> >>>> >>>
>> >>>> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
>> >>>> >>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>> >>>> >>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>> >>>> >>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>> >>>> >>>    zookeeper-3.3.1.jar
>> >>>> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
>> >>>> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
>> >>>> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
>> >>>> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
>> >>>> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
>> >>>> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
>> >>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
>> >>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
>> >>>> >>> Estimated value of PI is 3.1534
>> >>>> >>> Job Finished in 7.886 seconds
>> >>>> >>> edward@slave:~/workspace/hama-trunk$
>> >>>> >>>
>> >>>> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
>> >>>> >>> <jo...@cbsinteractive.com> wrote:
>> >>>> >>>> Replying to my own post ...
>> >>>> >>>>
>> >>>> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>> >>>> >>>> <jo...@cbsinteractive.com> wrote:
>> >>>> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>> >>>> >>>>>>
>> >>>> >>>>>> When I try to run the hama daemons using the command -
>> >>>> >>>>>> bin/hama zookeeper
>> >>>> >>>>>>
>> >>>> >>>>>> I get this error -
>> >>>> >>>>
>> >>>> >>>> It looks like the problem is actually this line in pom.xml (line 205)  :
>> >>>> >>>>                    <excludeTransitive>true</excludeTransitive>
>> >>>> >>>>
>> >>>> >>>> This prevents commons-cli (and other transitive dependencies) from
>> >>>> >>>> being copied to the lib dir like they should be.
>> >>>> >>>>
>> >>>> >>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
>> >>>> >>>> JIRA for this after lunch, the command line is fairly broken with this
>> >>>> >>>> line in, especially after several dependencies were (correctly)
>> >>>> >>>> removed in HAMA-402.
>> >>>> >>>>
>> >>>> >>>>
>> >>>> >>>> ...joe
>> >>>> >>>>
>> >>>> >>>>
>> >>>> >>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
>> >>>> >>>>> commons-cli dependency was removed recently as part of HAMA-402, and
>> >>>> >>>>> re-adding it fixes the NoClassDefFoundError.
>> >>>> >>>>>
>> >>>> >>>>> I believe commons-cli should have been pulled as part of the
>> >>>> >>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
>> >>>> >>>>> rusty, and I'm no longer sure just how transitive dependencies are
>> >>>> >>>>> pulled)
>> >>>> >>>>>
>> >>>> >>>>> While a quick work-around is to re-add the dependency here, the proper
>> >>>> >>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
>> >>>> >>>>> depends on commons-cli.
>> >>>> >>>>>
>> >>>> >>>>>
>> >>>> >>>>> ...joe
>> >>>> >>>>>
>> >>>> >>>>>
>> >>>> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> >>>> >>>>>> org/apache/commons/cli/ParseException
>> >>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>> >>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>> >>>> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>> >>>> >>>>>> Caused by: java.lang.ClassNotFoundException:
>> >>>> >>>>>> org.apache.commons.cli.ParseException
>> >>>> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> >>>> >>>>>> at java.security.AccessController.doPrivileged(Native Method)
>> >>>> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> >>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> >>>> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> >>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> >>>> >>>>>> ... 3 more
>> >>>> >>>>>>
>> >>>> >>>>>>
>> >>>> >>>>>> I tried installing it a few times, but can't think of a solution to the
>> >>>> >>>>>> problem. Can you please help ?
>> >>>> >>>>>>
>> >>>> >>>>>> Thank you
>> >>>> >>>>>>
>> >>>> >>>>>> Sincerely,
>> >>>> >>>>>> Ashish
>> >>>> >>>>>>
>> >>>> >>>>>>
>> >>>> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>> >>>> >>>>>>
>> >>>> >>>>>>> Hi,
>> >>>> >>>>>>>
>> >>>> >>>>>>> I would suggest following steps to debug:
>> >>>> >>>>>>>
>> >>>> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>> >>>> >>>>>>> 2. Run the Hama deamons with 3 terminals.
>> >>>> >>>>>>>
>> >>>> >>>>>>> % bin/hama zookeeper
>> >>>> >>>>>>> % bin/hama bspmaster
>> >>>> >>>>>>> % bin/hama groom
>> >>>> >>>>>>>
>> >>>> >>>>>>> 3. Use the logging API to log debug.
>> >>>> >>>>>>> For example, LOG.debug("like this");
>> >>>> >>>>>>>
>> >>>> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>> >>>> >>>>>>> wrote:
>> >>>> >>>>>>> > Hi,
>> >>>> >>>>>>> >
>> >>>> >>>>>>> > I was wondering if anyone can give me instructions on how to debug code
>> >>>> >>>>>>> in
>> >>>> >>>>>>> > distributed or pseudo-distributed mode ? It would really help me.
>> >>>> >>>>>>> >
>> >>>> >>>>>>> > Thank you
>> >>>> >>>>>>> >
>> >>>> >>>>>>> > Sincerely,
>> >>>> >>>>>>> > Ashish
>> >>>> >>>>>>> >
>> >>>> >>>>>>> >
>> >>>> >>>>>>> >
>> >>>> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>> >>>> >>>>>>> >wrote:
>> >>>> >>>>>>> >
>> >>>> >>>>>>> >>
>> >>>> >>>>>>> >> Hi,
>> >>>> >>>>>>> >>
>> >>>> >>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>> >>>> >>>>>>> to
>> >>>> >>>>>>> >> debug.
>> >>>> >>>>>>> >>
>> >>>> >>>>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
>> >>>> >>>>>>> can
>> >>>> >>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>> >>>> >>>>>>> be
>> >>>> >>>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>> >>>> >>>>>>> >> server), but where do I specify that in the code in eclipse ?
>> >>>> >>>>>>> >>
>> >>>> >>>>>>> >> My next question is - what is the difference between MiniBSPCluster and
>> >>>> >>>>>>> >> LocalBSPRunner  and which file should I be using to be able to step
>> >>>> >>>>>>> through
>> >>>> >>>>>>> >> the code ?
>> >>>> >>>>>>> >>
>> >>>> >>>>>>> >> These are the processes currently running on my machine -
>> >>>> >>>>>>> >>
>> >>>> >>>>>>> >> 6112 NameNode
>> >>>> >>>>>>> >> 3319 ZooKeeperRunner
>> >>>> >>>>>>> >> 3018
>> >>>> >>>>>>> >> 6449 SecondaryNameNode
>> >>>> >>>>>>> >> 5709 HMaster
>> >>>> >>>>>>> >> 5643 HQuorumPeer
>> >>>> >>>>>>> >> 6684 TaskTracker
>> >>>> >>>>>>> >> 6526 JobTracker
>> >>>> >>>>>>> >> 8074 Jps
>> >>>> >>>>>>> >> 6277 DataNode
>> >>>> >>>>>>> >>
>> >>>> >>>>>>> >> Thank you,
>> >>>> >>>>>>> >> Ashish
>> >>>> >>>>>>> >>
>> >>>> >>>>>>> >
>> >>>> >>>>>>>
>> >>>> >>>>>>>
>> >>>> >>>>>>>
>> >>>> >>>>>>> --
>> >>>> >>>>>>> Best Regards, Edward J. Yoon
>> >>>> >>>>>>> @eddieyoon
>> >>>> >>>>>>>
>> >>>> >>>>>>
>> >>>> >>>>>
>> >>>> >>>>
>> >>>> >>>
>> >>>> >>>
>> >>>> >>>
>> >>>> >>> --
>> >>>> >>> Best Regards, Edward J. Yoon
>> >>>> >>> @eddieyoon
>> >>>> >>>
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> --
>> >>>> >> Best Regards, Edward J. Yoon
>> >>>> >> @eddieyoon
>> >>>> >>
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > --
>> >>>> > Best Regards, Edward J. Yoon
>> >>>> > @eddieyoon
>> >>>> >
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards, Edward J. Yoon
>> >> @eddieyoon
>> >>
>> >
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: Debugging hama in eclipse

Posted by Ashish Agarwal <as...@gmail.com>.

I already have made that change, but I still have the same error

I have tried
<value>localhost:40000</value>
and also
<value>local</value>


Thanks
Ashish



On Thu, Jun 23, 2011 at 1:12 AM, Edward J. Yoon <ed...@apache.org> wrote:
>
> You need to set the 'bsp.master.address' to localhost or your hostname.
>
> conf/hama-site.xml:
>
>  <property>
>    <name>bsp.master.address</name>
>    <value>localhost:40000</value>
>  </property>
>
> On Thu, Jun 23, 2011 at 5:01 PM, Ashish Agarwal <as...@gmail.com> wrote:
> > Now, I am having a problem starting the bspmaster :(
> > bin/hama bspmaster
> >
> >
> > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama bspmaster
> > 11/06/23 01:00:13 INFO hama.BSPMasterRunner: STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting BSPMaster
> > STARTUP_MSG:   host = ubuntu/127.0.1.1
> > STARTUP_MSG:   args = []
> > STARTUP_MSG:   version = 0.20.2
> > STARTUP_MSG:   build =  -r ; compiled by 'chrisdo' on Wed Mar 10
> > 13:22:54 PST 2010
> > ************************************************************/
> > 11/06/23 01:00:13 INFO bsp.BSPMaster: RPC BSPMaster: host local port 40000
> > 11/06/23 01:00:14 FATAL hama.BSPMasterRunner:
> > java.net.UnknownHostException: Invalid hostname for server: local
> >        at org.apache.hadoop.ipc.Server.bind(Server.java:198)
> >        at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
> >        at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
> >        at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
> >        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
> >        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:441)
> >        at org.apache.hama.bsp.BSPMaster.<init>(BSPMaster.java:248)
> >        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:419)
> >        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:413)
> >        at org.apache.hama.BSPMasterRunner.run(BSPMasterRunner.java:46)
> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> >        at org.apache.hama.BSPMasterRunner.main(BSPMasterRunner.java:56)
> >
> > 11/06/23 01:00:14 INFO hama.BSPMasterRunner: SHUTDOWN_MSG:
> > /************************************************************
> > SHUTDOWN_MSG: Shutting down BSPMaster at ubuntu/127.0.1.1
> > ************************************************************/
> > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
> >
> >
> > Thanks
> > Ashish
> >
> > On Thu, Jun 23, 2011 at 12:49 AM, Edward J. Yoon <ed...@apache.org> wrote:
> >> If you want to execute a command in the background without watching
> >> logs, you can do the following:
> >>
> >> % bin/hama zookeeper &
> >>
> >> On Thu, Jun 23, 2011 at 4:43 PM, Ashish Agarwal <as...@gmail.com> wrote:
> >>> How long does it take to execute the command
> >>> bin/hama zookeeper
> >>>
> >>> I had to stop the execution as it stopped doing anything after some time.
> >>>
> >>> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> >>> environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
> >>> GMT
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> >>> environment:host.name=ubuntu.ubuntu-domain
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> >>> environment:java.version=1.6.0_24
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> >>> environment:java.vendor=Sun Microsystems Inc.
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> >>> environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> >>> environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> >>> environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> >>> environment:java.io.tmpdir=/tmp
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> >>> environment:java.compiler=<NA>
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.name=Linux
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.arch=i386
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> >>> environment:os.version=2.6.35-28-generic
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> >>> environment:user.name=hadoop
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> >>> environment:user.home=/home/hadoop
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> >>> environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1
> >>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to 1200000
> >>> 11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port
> >>> 0.0.0.0/0.0.0.0:21810
> >>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
> >>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
> >>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
> >>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
> >>> 11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0
> >>> ^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
> >>>
> >>>
> >>> Thanks
> >>> Ashish
> >>>
> >>>
> >>>
> >>> On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd
> >>> <jo...@cbsinteractive.com> wrote:
> >>>>
> >>>> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <ed...@apache.org> wrote:
> >>>> > Instead removing 206 line, why don't we add dependencies as below?
> >>>>
> >>>> I think removing line 206 is the right thing to do.  Hama doesn't
> >>>> depend on commons-cli, hadoop-core does, and maven will properly pick
> >>>> that up and include the jar if allowed.
> >>>>
> >>>> The fact that Hama doesn't depend on the several items below was
> >>>> correctly detected by maven, in the report description of HAMA-402.
> >>>>
> >>>> You don't mention a particular problem with including the transitive
> >>>> dependencies, but if your problem is that it includes too many items,
> >>>> maven does have options to exclude a particular transitive dependency
> >>>> on a case-by-case basis that might be more appropriate to use.
> >>>>
> >>>> Including the dependencies below as you suggest puts Hama in the place
> >>>> of maintaining a transitive dependency like commons-cli in lock-step
> >>>> hadoop-core, where the actual dependency comes from.
> >>>>
> >>>>
> >>>> This is all just my $.02 worth, I'm open to arguments in favor of
> >>>> <excludeTransitive/>, and there is often more than one way to skin a
> >>>> cat with respect to dependency management.
> >>>>
> >>>>
> >>>> ...joe
> >>>>
> >>>>
> >>>>
> >>>> >    <dependency>
> >>>> >      <groupId>org.mortbay.jetty</groupId>
> >>>> >      <artifactId>jetty-annotations</artifactId>
> >>>> >      <version>6.1H.22</version>
> >>>> >    </dependency>
> >>>> >    <dependency>
> >>>> >      <groupId>org.slf4j</groupId>
> >>>> >      <artifactId>slf4j-log4j12</artifactId>
> >>>> >      <version>1.5.2</version>
> >>>> >    </dependency>
> >>>> >    <dependency>
> >>>> >      <groupId>commons-cli</groupId>
> >>>> >      <artifactId>commons-cli</artifactId>
> >>>> >      <version>1.2</version>
> >>>> >    </dependency>
> >>>> >
> >>>> > Thanks.
> >>>> >
> >>>> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <ed...@apache.org> wrote:
> >>>> >> Oh... sorry, you're right.
> >>>> >>
> >>>> >> It looks like a bug. We have to fix this.
> >>>> >>
> >>>> >> [INFO] Total time: 4:36.098s
> >>>> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
> >>>> >> [INFO] Final Memory: 20M/51M
> >>>> >> [INFO] ------------------------------------------------------------------------
> >>>> >> edward@test ~/workspace/hama-trunk $ ls lib/
> >>>> >> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
> >>>> >> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
> >>>> >> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
> >>>> >>    zookeeper-3.3.1.jar
> >>>> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
> >>>> >> -bash: bin/hama: Permission denied
> >>>> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
> >>>> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
> >>>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
> >>>> >> org/apache/commons/cli/ParseException
> >>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
> >>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> >>>> >>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
> >>>> >> Caused by: java.lang.ClassNotFoundException:
> >>>> >> org.apache.commons.cli.ParseException
> >>>> >>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> >>>> >>        at java.security.AccessController.doPrivileged(Native Method)
> >>>> >>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> >>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
> >>>> >>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >>>> >>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
> >>>> >>        ... 3 more
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <ed...@apache.org> wrote:
> >>>> >>> That's so weird.
> >>>> >>>
> >>>> >>> It works well on my test cluster.
> >>>> >>>
> >>>> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
> >>>> >>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
> >>>> >>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
> >>>> >>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
> >>>> >>>    zookeeper-3.3.1.jar
> >>>> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
> >>>> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
> >>>> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
> >>>> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
> >>>> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
> >>>> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
> >>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
> >>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
> >>>> >>> Estimated value of PI is 3.1534
> >>>> >>> Job Finished in 7.886 seconds
> >>>> >>> edward@slave:~/workspace/hama-trunk$
> >>>> >>>
> >>>> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
> >>>> >>> <jo...@cbsinteractive.com> wrote:
> >>>> >>>> Replying to my own post ...
> >>>> >>>>
> >>>> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
> >>>> >>>> <jo...@cbsinteractive.com> wrote:
> >>>> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
> >>>> >>>>>>
> >>>> >>>>>> When I try to run the hama daemons using the command -
> >>>> >>>>>> bin/hama zookeeper
> >>>> >>>>>>
> >>>> >>>>>> I get this error -
> >>>> >>>>
> >>>> >>>> It looks like the problem is actually this line in pom.xml (line 205)  :
> >>>> >>>>                    <excludeTransitive>true</excludeTransitive>
> >>>> >>>>
> >>>> >>>> This prevents commons-cli (and other transitive dependencies) from
> >>>> >>>> being copied to the lib dir like they should be.
> >>>> >>>>
> >>>> >>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
> >>>> >>>> JIRA for this after lunch, the command line is fairly broken with this
> >>>> >>>> line in, especially after several dependencies were (correctly)
> >>>> >>>> removed in HAMA-402.
> >>>> >>>>
> >>>> >>>>
> >>>> >>>> ...joe
> >>>> >>>>
> >>>> >>>>
> >>>> >>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
> >>>> >>>>> commons-cli dependency was removed recently as part of HAMA-402, and
> >>>> >>>>> re-adding it fixes the NoClassDefFoundError.
> >>>> >>>>>
> >>>> >>>>> I believe commons-cli should have been pulled as part of the
> >>>> >>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
> >>>> >>>>> rusty, and I'm no longer sure just how transitive dependencies are
> >>>> >>>>> pulled)
> >>>> >>>>>
> >>>> >>>>> While a quick work-around is to re-add the dependency here, the proper
> >>>> >>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
> >>>> >>>>> depends on commons-cli.
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>> ...joe
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
> >>>> >>>>>> org/apache/commons/cli/ParseException
> >>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
> >>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> >>>> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
> >>>> >>>>>> Caused by: java.lang.ClassNotFoundException:
> >>>> >>>>>> org.apache.commons.cli.ParseException
> >>>> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>>> >>>>>> at java.security.AccessController.doPrivileged(Native Method)
> >>>> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >>>> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >>>> >>>>>> ... 3 more
> >>>> >>>>>>
> >>>> >>>>>>
> >>>> >>>>>> I tried installing it a few times, but can't think of a solution to the
> >>>> >>>>>> problem. Can you please help ?
> >>>> >>>>>>
> >>>> >>>>>> Thank you
> >>>> >>>>>>
> >>>> >>>>>> Sincerely,
> >>>> >>>>>> Ashish
> >>>> >>>>>>
> >>>> >>>>>>
> >>>> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
> >>>> >>>>>>
> >>>> >>>>>>> Hi,
> >>>> >>>>>>>
> >>>> >>>>>>> I would suggest following steps to debug:
> >>>> >>>>>>>
> >>>> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
> >>>> >>>>>>> 2. Run the Hama deamons with 3 terminals.
> >>>> >>>>>>>
> >>>> >>>>>>> % bin/hama zookeeper
> >>>> >>>>>>> % bin/hama bspmaster
> >>>> >>>>>>> % bin/hama groom
> >>>> >>>>>>>
> >>>> >>>>>>> 3. Use the logging API to log debug.
> >>>> >>>>>>> For example, LOG.debug("like this");
> >>>> >>>>>>>
> >>>> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
> >>>> >>>>>>> wrote:
> >>>> >>>>>>> > Hi,
> >>>> >>>>>>> >
> >>>> >>>>>>> > I was wondering if anyone can give me instructions on how to debug code
> >>>> >>>>>>> in
> >>>> >>>>>>> > distributed or pseudo-distributed mode ? It would really help me.
> >>>> >>>>>>> >
> >>>> >>>>>>> > Thank you
> >>>> >>>>>>> >
> >>>> >>>>>>> > Sincerely,
> >>>> >>>>>>> > Ashish
> >>>> >>>>>>> >
> >>>> >>>>>>> >
> >>>> >>>>>>> >
> >>>> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
> >>>> >>>>>>> >wrote:
> >>>> >>>>>>> >
> >>>> >>>>>>> >>
> >>>> >>>>>>> >> Hi,
> >>>> >>>>>>> >>
> >>>> >>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
> >>>> >>>>>>> to
> >>>> >>>>>>> >> debug.
> >>>> >>>>>>> >>
> >>>> >>>>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
> >>>> >>>>>>> can
> >>>> >>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
> >>>> >>>>>>> be
> >>>> >>>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
> >>>> >>>>>>> >> server), but where do I specify that in the code in eclipse ?
> >>>> >>>>>>> >>
> >>>> >>>>>>> >> My next question is - what is the difference between MiniBSPCluster and
> >>>> >>>>>>> >> LocalBSPRunner  and which file should I be using to be able to step
> >>>> >>>>>>> through
> >>>> >>>>>>> >> the code ?
> >>>> >>>>>>> >>
> >>>> >>>>>>> >> These are the processes currently running on my machine -
> >>>> >>>>>>> >>
> >>>> >>>>>>> >> 6112 NameNode
> >>>> >>>>>>> >> 3319 ZooKeeperRunner
> >>>> >>>>>>> >> 3018
> >>>> >>>>>>> >> 6449 SecondaryNameNode
> >>>> >>>>>>> >> 5709 HMaster
> >>>> >>>>>>> >> 5643 HQuorumPeer
> >>>> >>>>>>> >> 6684 TaskTracker
> >>>> >>>>>>> >> 6526 JobTracker
> >>>> >>>>>>> >> 8074 Jps
> >>>> >>>>>>> >> 6277 DataNode
> >>>> >>>>>>> >>
> >>>> >>>>>>> >> Thank you,
> >>>> >>>>>>> >> Ashish
> >>>> >>>>>>> >>
> >>>> >>>>>>> >
> >>>> >>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>> --
> >>>> >>>>>>> Best Regards, Edward J. Yoon
> >>>> >>>>>>> @eddieyoon
> >>>> >>>>>>>
> >>>> >>>>>>
> >>>> >>>>>
> >>>> >>>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>> --
> >>>> >>> Best Regards, Edward J. Yoon
> >>>> >>> @eddieyoon
> >>>> >>>
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> --
> >>>> >> Best Regards, Edward J. Yoon
> >>>> >> @eddieyoon
> >>>> >>
> >>>> >
> >>>> >
> >>>> >
> >>>> > --
> >>>> > Best Regards, Edward J. Yoon
> >>>> > @eddieyoon
> >>>> >
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> @eddieyoon
> >>
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon

Re: Debugging hama in eclipse

Posted by "Edward J. Yoon" <ed...@apache.org>.

You need to set the 'bsp.master.address' to localhost or your hostname.

conf/hama-site.xml:

  <property>
    <name>bsp.master.address</name>
    <value>localhost:40000</value>
  </property>

On Thu, Jun 23, 2011 at 5:01 PM, Ashish Agarwal <as...@gmail.com> wrote:
> Now, I am having a problem starting the bspmaster :(
> bin/hama bspmaster
>
>
> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama bspmaster
> 11/06/23 01:00:13 INFO hama.BSPMasterRunner: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting BSPMaster
> STARTUP_MSG:   host = ubuntu/127.0.1.1
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.2
> STARTUP_MSG:   build =  -r ; compiled by 'chrisdo' on Wed Mar 10
> 13:22:54 PST 2010
> ************************************************************/
> 11/06/23 01:00:13 INFO bsp.BSPMaster: RPC BSPMaster: host local port 40000
> 11/06/23 01:00:14 FATAL hama.BSPMasterRunner:
> java.net.UnknownHostException: Invalid hostname for server: local
>        at org.apache.hadoop.ipc.Server.bind(Server.java:198)
>        at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
>        at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
>        at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
>        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
>        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:441)
>        at org.apache.hama.bsp.BSPMaster.<init>(BSPMaster.java:248)
>        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:419)
>        at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:413)
>        at org.apache.hama.BSPMasterRunner.run(BSPMasterRunner.java:46)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>        at org.apache.hama.BSPMasterRunner.main(BSPMasterRunner.java:56)
>
> 11/06/23 01:00:14 INFO hama.BSPMasterRunner: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down BSPMaster at ubuntu/127.0.1.1
> ************************************************************/
> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>
>
> Thanks
> Ashish
>
> On Thu, Jun 23, 2011 at 12:49 AM, Edward J. Yoon <ed...@apache.org> wrote:
>> If you want to execute a command in the background without watching
>> logs, you can do the following:
>>
>> % bin/hama zookeeper &
>>
>> On Thu, Jun 23, 2011 at 4:43 PM, Ashish Agarwal <as...@gmail.com> wrote:
>>> How long does it take to execute the command
>>> bin/hama zookeeper
>>>
>>> I had to stop the execution as it stopped doing anything after some time.
>>>
>>> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
>>> GMT
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:host.name=ubuntu.ubuntu-domain
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:java.version=1.6.0_24
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:java.vendor=Sun Microsystems Inc.
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:java.io.tmpdir=/tmp
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:java.compiler=<NA>
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.name=Linux
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.arch=i386
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:os.version=2.6.35-28-generic
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:user.name=hadoop
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:user.home=/home/hadoop
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>>> environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1
>>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to 1200000
>>> 11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port
>>> 0.0.0.0/0.0.0.0:21810
>>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>>> 11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0
>>> ^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>>>
>>>
>>> Thanks
>>> Ashish
>>>
>>>
>>>
>>> On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd
>>> <jo...@cbsinteractive.com> wrote:
>>>>
>>>> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <ed...@apache.org> wrote:
>>>> > Instead removing 206 line, why don't we add dependencies as below?
>>>>
>>>> I think removing line 206 is the right thing to do.  Hama doesn't
>>>> depend on commons-cli, hadoop-core does, and maven will properly pick
>>>> that up and include the jar if allowed.
>>>>
>>>> The fact that Hama doesn't depend on the several items below was
>>>> correctly detected by maven, in the report description of HAMA-402.
>>>>
>>>> You don't mention a particular problem with including the transitive
>>>> dependencies, but if your problem is that it includes too many items,
>>>> maven does have options to exclude a particular transitive dependency
>>>> on a case-by-case basis that might be more appropriate to use.
>>>>
>>>> Including the dependencies below as you suggest puts Hama in the place
>>>> of maintaining a transitive dependency like commons-cli in lock-step
>>>> hadoop-core, where the actual dependency comes from.
>>>>
>>>>
>>>> This is all just my $.02 worth, I'm open to arguments in favor of
>>>> <excludeTransitive/>, and there is often more than one way to skin a
>>>> cat with respect to dependency management.
>>>>
>>>>
>>>> ...joe
>>>>
>>>>
>>>>
>>>> >    <dependency>
>>>> >      <groupId>org.mortbay.jetty</groupId>
>>>> >      <artifactId>jetty-annotations</artifactId>
>>>> >      <version>6.1H.22</version>
>>>> >    </dependency>
>>>> >    <dependency>
>>>> >      <groupId>org.slf4j</groupId>
>>>> >      <artifactId>slf4j-log4j12</artifactId>
>>>> >      <version>1.5.2</version>
>>>> >    </dependency>
>>>> >    <dependency>
>>>> >      <groupId>commons-cli</groupId>
>>>> >      <artifactId>commons-cli</artifactId>
>>>> >      <version>1.2</version>
>>>> >    </dependency>
>>>> >
>>>> > Thanks.
>>>> >
>>>> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>> >> Oh... sorry, you're right.
>>>> >>
>>>> >> It looks like a bug. We have to fix this.
>>>> >>
>>>> >> [INFO] Total time: 4:36.098s
>>>> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
>>>> >> [INFO] Final Memory: 20M/51M
>>>> >> [INFO] ------------------------------------------------------------------------
>>>> >> edward@test ~/workspace/hama-trunk $ ls lib/
>>>> >> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>>> >> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>>> >> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>>> >>    zookeeper-3.3.1.jar
>>>> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
>>>> >> -bash: bin/hama: Permission denied
>>>> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
>>>> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
>>>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> >> org/apache/commons/cli/ParseException
>>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>> >>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>> >> Caused by: java.lang.ClassNotFoundException:
>>>> >> org.apache.commons.cli.ParseException
>>>> >>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>> >>        at java.security.AccessController.doPrivileged(Native Method)
>>>> >>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
>>>> >>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>> >>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
>>>> >>        ... 3 more
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>> >>> That's so weird.
>>>> >>>
>>>> >>> It works well on my test cluster.
>>>> >>>
>>>> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
>>>> >>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>>> >>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>>> >>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>>> >>>    zookeeper-3.3.1.jar
>>>> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
>>>> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
>>>> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
>>>> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
>>>> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
>>>> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
>>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
>>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
>>>> >>> Estimated value of PI is 3.1534
>>>> >>> Job Finished in 7.886 seconds
>>>> >>> edward@slave:~/workspace/hama-trunk$
>>>> >>>
>>>> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
>>>> >>> <jo...@cbsinteractive.com> wrote:
>>>> >>>> Replying to my own post ...
>>>> >>>>
>>>> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>>>> >>>> <jo...@cbsinteractive.com> wrote:
>>>> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>>>> >>>>>>
>>>> >>>>>> When I try to run the hama daemons using the command -
>>>> >>>>>> bin/hama zookeeper
>>>> >>>>>>
>>>> >>>>>> I get this error -
>>>> >>>>
>>>> >>>> It looks like the problem is actually this line in pom.xml (line 205)  :
>>>> >>>>                    <excludeTransitive>true</excludeTransitive>
>>>> >>>>
>>>> >>>> This prevents commons-cli (and other transitive dependencies) from
>>>> >>>> being copied to the lib dir like they should be.
>>>> >>>>
>>>> >>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
>>>> >>>> JIRA for this after lunch, the command line is fairly broken with this
>>>> >>>> line in, especially after several dependencies were (correctly)
>>>> >>>> removed in HAMA-402.
>>>> >>>>
>>>> >>>>
>>>> >>>> ...joe
>>>> >>>>
>>>> >>>>
>>>> >>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
>>>> >>>>> commons-cli dependency was removed recently as part of HAMA-402, and
>>>> >>>>> re-adding it fixes the NoClassDefFoundError.
>>>> >>>>>
>>>> >>>>> I believe commons-cli should have been pulled as part of the
>>>> >>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
>>>> >>>>> rusty, and I'm no longer sure just how transitive dependencies are
>>>> >>>>> pulled)
>>>> >>>>>
>>>> >>>>> While a quick work-around is to re-add the dependency here, the proper
>>>> >>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
>>>> >>>>> depends on commons-cli.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> ...joe
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> >>>>>> org/apache/commons/cli/ParseException
>>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>> >>>>>> Caused by: java.lang.ClassNotFoundException:
>>>> >>>>>> org.apache.commons.cli.ParseException
>>>> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> >>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>> >>>>>> ... 3 more
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> I tried installing it a few times, but can't think of a solution to the
>>>> >>>>>> problem. Can you please help ?
>>>> >>>>>>
>>>> >>>>>> Thank you
>>>> >>>>>>
>>>> >>>>>> Sincerely,
>>>> >>>>>> Ashish
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>>>> >>>>>>
>>>> >>>>>>> Hi,
>>>> >>>>>>>
>>>> >>>>>>> I would suggest following steps to debug:
>>>> >>>>>>>
>>>> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>>>> >>>>>>> 2. Run the Hama deamons with 3 terminals.
>>>> >>>>>>>
>>>> >>>>>>> % bin/hama zookeeper
>>>> >>>>>>> % bin/hama bspmaster
>>>> >>>>>>> % bin/hama groom
>>>> >>>>>>>
>>>> >>>>>>> 3. Use the logging API to log debug.
>>>> >>>>>>> For example, LOG.debug("like this");
>>>> >>>>>>>
>>>> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>>>> >>>>>>> wrote:
>>>> >>>>>>> > Hi,
>>>> >>>>>>> >
>>>> >>>>>>> > I was wondering if anyone can give me instructions on how to debug code
>>>> >>>>>>> in
>>>> >>>>>>> > distributed or pseudo-distributed mode ? It would really help me.
>>>> >>>>>>> >
>>>> >>>>>>> > Thank you
>>>> >>>>>>> >
>>>> >>>>>>> > Sincerely,
>>>> >>>>>>> > Ashish
>>>> >>>>>>> >
>>>> >>>>>>> >
>>>> >>>>>>> >
>>>> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>>>> >>>>>>> >wrote:
>>>> >>>>>>> >
>>>> >>>>>>> >>
>>>> >>>>>>> >> Hi,
>>>> >>>>>>> >>
>>>> >>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>>>> >>>>>>> to
>>>> >>>>>>> >> debug.
>>>> >>>>>>> >>
>>>> >>>>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
>>>> >>>>>>> can
>>>> >>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>>>> >>>>>>> be
>>>> >>>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>>>> >>>>>>> >> server), but where do I specify that in the code in eclipse ?
>>>> >>>>>>> >>
>>>> >>>>>>> >> My next question is - what is the difference between MiniBSPCluster and
>>>> >>>>>>> >> LocalBSPRunner  and which file should I be using to be able to step
>>>> >>>>>>> through
>>>> >>>>>>> >> the code ?
>>>> >>>>>>> >>
>>>> >>>>>>> >> These are the processes currently running on my machine -
>>>> >>>>>>> >>
>>>> >>>>>>> >> 6112 NameNode
>>>> >>>>>>> >> 3319 ZooKeeperRunner
>>>> >>>>>>> >> 3018
>>>> >>>>>>> >> 6449 SecondaryNameNode
>>>> >>>>>>> >> 5709 HMaster
>>>> >>>>>>> >> 5643 HQuorumPeer
>>>> >>>>>>> >> 6684 TaskTracker
>>>> >>>>>>> >> 6526 JobTracker
>>>> >>>>>>> >> 8074 Jps
>>>> >>>>>>> >> 6277 DataNode
>>>> >>>>>>> >>
>>>> >>>>>>> >> Thank you,
>>>> >>>>>>> >> Ashish
>>>> >>>>>>> >>
>>>> >>>>>>> >
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> --
>>>> >>>>>>> Best Regards, Edward J. Yoon
>>>> >>>>>>> @eddieyoon
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>>
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Best Regards, Edward J. Yoon
>>>> >>> @eddieyoon
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Best Regards, Edward J. Yoon
>>>> >> @eddieyoon
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Best Regards, Edward J. Yoon
>>>> > @eddieyoon
>>>> >
>>>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: Debugging hama in eclipse

Posted by Ashish Agarwal <as...@gmail.com>.

Now, I am having a problem starting the bspmaster :(
bin/hama bspmaster


hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama bspmaster
11/06/23 01:00:13 INFO hama.BSPMasterRunner: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting BSPMaster
STARTUP_MSG:   host = ubuntu/127.0.1.1
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build =  -r ; compiled by 'chrisdo' on Wed Mar 10
13:22:54 PST 2010
************************************************************/
11/06/23 01:00:13 INFO bsp.BSPMaster: RPC BSPMaster: host local port 40000
11/06/23 01:00:14 FATAL hama.BSPMasterRunner:
java.net.UnknownHostException: Invalid hostname for server: local
	at org.apache.hadoop.ipc.Server.bind(Server.java:198)
	at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
	at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
	at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
	at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
	at org.apache.hadoop.ipc.RPC.getServer(RPC.java:441)
	at org.apache.hama.bsp.BSPMaster.<init>(BSPMaster.java:248)
	at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:419)
	at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:413)
	at org.apache.hama.BSPMasterRunner.run(BSPMasterRunner.java:46)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at org.apache.hama.BSPMasterRunner.main(BSPMasterRunner.java:56)

11/06/23 01:00:14 INFO hama.BSPMasterRunner: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down BSPMaster at ubuntu/127.0.1.1
************************************************************/
hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$


Thanks
Ashish

On Thu, Jun 23, 2011 at 12:49 AM, Edward J. Yoon <ed...@apache.org> wrote:
> If you want to execute a command in the background without watching
> logs, you can do the following:
>
> % bin/hama zookeeper &
>
> On Thu, Jun 23, 2011 at 4:43 PM, Ashish Agarwal <as...@gmail.com> wrote:
>> How long does it take to execute the command
>> bin/hama zookeeper
>>
>> I had to stop the execution as it stopped doing anything after some time.
>>
>> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper
>> 11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
>> GMT
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:host.name=ubuntu.ubuntu-domain
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:java.version=1.6.0_24
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:java.vendor=Sun Microsystems Inc.
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:java.io.tmpdir=/tmp
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:java.compiler=<NA>
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.name=Linux
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.arch=i386
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:os.version=2.6.35-28-generic
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:user.name=hadoop
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:user.home=/home/hadoop
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to 1200000
>> 11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port
>> 0.0.0.0/0.0.0.0:21810
>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>> 11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0
>> ^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>>
>>
>> Thanks
>> Ashish
>>
>>
>>
>> On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd
>> <jo...@cbsinteractive.com> wrote:
>>>
>>> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <ed...@apache.org> wrote:
>>> > Instead removing 206 line, why don't we add dependencies as below?
>>>
>>> I think removing line 206 is the right thing to do.  Hama doesn't
>>> depend on commons-cli, hadoop-core does, and maven will properly pick
>>> that up and include the jar if allowed.
>>>
>>> The fact that Hama doesn't depend on the several items below was
>>> correctly detected by maven, in the report description of HAMA-402.
>>>
>>> You don't mention a particular problem with including the transitive
>>> dependencies, but if your problem is that it includes too many items,
>>> maven does have options to exclude a particular transitive dependency
>>> on a case-by-case basis that might be more appropriate to use.
>>>
>>> Including the dependencies below as you suggest puts Hama in the place
>>> of maintaining a transitive dependency like commons-cli in lock-step
>>> hadoop-core, where the actual dependency comes from.
>>>
>>>
>>> This is all just my $.02 worth, I'm open to arguments in favor of
>>> <excludeTransitive/>, and there is often more than one way to skin a
>>> cat with respect to dependency management.
>>>
>>>
>>> ...joe
>>>
>>>
>>>
>>> >    <dependency>
>>> >      <groupId>org.mortbay.jetty</groupId>
>>> >      <artifactId>jetty-annotations</artifactId>
>>> >      <version>6.1H.22</version>
>>> >    </dependency>
>>> >    <dependency>
>>> >      <groupId>org.slf4j</groupId>
>>> >      <artifactId>slf4j-log4j12</artifactId>
>>> >      <version>1.5.2</version>
>>> >    </dependency>
>>> >    <dependency>
>>> >      <groupId>commons-cli</groupId>
>>> >      <artifactId>commons-cli</artifactId>
>>> >      <version>1.2</version>
>>> >    </dependency>
>>> >
>>> > Thanks.
>>> >
>>> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>> >> Oh... sorry, you're right.
>>> >>
>>> >> It looks like a bug. We have to fix this.
>>> >>
>>> >> [INFO] Total time: 4:36.098s
>>> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
>>> >> [INFO] Final Memory: 20M/51M
>>> >> [INFO] ------------------------------------------------------------------------
>>> >> edward@test ~/workspace/hama-trunk $ ls lib/
>>> >> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>> >> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>> >> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>> >>    zookeeper-3.3.1.jar
>>> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
>>> >> -bash: bin/hama: Permission denied
>>> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
>>> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
>>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> >> org/apache/commons/cli/ParseException
>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>> >>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>> >> Caused by: java.lang.ClassNotFoundException:
>>> >> org.apache.commons.cli.ParseException
>>> >>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>> >>        at java.security.AccessController.doPrivileged(Native Method)
>>> >>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
>>> >>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> >>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
>>> >>        ... 3 more
>>> >>
>>> >>
>>> >>
>>> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>> >>> That's so weird.
>>> >>>
>>> >>> It works well on my test cluster.
>>> >>>
>>> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
>>> >>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>> >>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>> >>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>> >>>    zookeeper-3.3.1.jar
>>> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
>>> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
>>> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
>>> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
>>> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
>>> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
>>> >>> Estimated value of PI is 3.1534
>>> >>> Job Finished in 7.886 seconds
>>> >>> edward@slave:~/workspace/hama-trunk$
>>> >>>
>>> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
>>> >>> <jo...@cbsinteractive.com> wrote:
>>> >>>> Replying to my own post ...
>>> >>>>
>>> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>>> >>>> <jo...@cbsinteractive.com> wrote:
>>> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>>> >>>>>>
>>> >>>>>> When I try to run the hama daemons using the command -
>>> >>>>>> bin/hama zookeeper
>>> >>>>>>
>>> >>>>>> I get this error -
>>> >>>>
>>> >>>> It looks like the problem is actually this line in pom.xml (line 205)  :
>>> >>>>                    <excludeTransitive>true</excludeTransitive>
>>> >>>>
>>> >>>> This prevents commons-cli (and other transitive dependencies) from
>>> >>>> being copied to the lib dir like they should be.
>>> >>>>
>>> >>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
>>> >>>> JIRA for this after lunch, the command line is fairly broken with this
>>> >>>> line in, especially after several dependencies were (correctly)
>>> >>>> removed in HAMA-402.
>>> >>>>
>>> >>>>
>>> >>>> ...joe
>>> >>>>
>>> >>>>
>>> >>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
>>> >>>>> commons-cli dependency was removed recently as part of HAMA-402, and
>>> >>>>> re-adding it fixes the NoClassDefFoundError.
>>> >>>>>
>>> >>>>> I believe commons-cli should have been pulled as part of the
>>> >>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
>>> >>>>> rusty, and I'm no longer sure just how transitive dependencies are
>>> >>>>> pulled)
>>> >>>>>
>>> >>>>> While a quick work-around is to re-add the dependency here, the proper
>>> >>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
>>> >>>>> depends on commons-cli.
>>> >>>>>
>>> >>>>>
>>> >>>>> ...joe
>>> >>>>>
>>> >>>>>
>>> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> >>>>>> org/apache/commons/cli/ParseException
>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>> >>>>>> Caused by: java.lang.ClassNotFoundException:
>>> >>>>>> org.apache.commons.cli.ParseException
>>> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> >>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> >>>>>> ... 3 more
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> I tried installing it a few times, but can't think of a solution to the
>>> >>>>>> problem. Can you please help ?
>>> >>>>>>
>>> >>>>>> Thank you
>>> >>>>>>
>>> >>>>>> Sincerely,
>>> >>>>>> Ashish
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>>> >>>>>>
>>> >>>>>>> Hi,
>>> >>>>>>>
>>> >>>>>>> I would suggest following steps to debug:
>>> >>>>>>>
>>> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>>> >>>>>>> 2. Run the Hama deamons with 3 terminals.
>>> >>>>>>>
>>> >>>>>>> % bin/hama zookeeper
>>> >>>>>>> % bin/hama bspmaster
>>> >>>>>>> % bin/hama groom
>>> >>>>>>>
>>> >>>>>>> 3. Use the logging API to log debug.
>>> >>>>>>> For example, LOG.debug("like this");
>>> >>>>>>>
>>> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>>> >>>>>>> wrote:
>>> >>>>>>> > Hi,
>>> >>>>>>> >
>>> >>>>>>> > I was wondering if anyone can give me instructions on how to debug code
>>> >>>>>>> in
>>> >>>>>>> > distributed or pseudo-distributed mode ? It would really help me.
>>> >>>>>>> >
>>> >>>>>>> > Thank you
>>> >>>>>>> >
>>> >>>>>>> > Sincerely,
>>> >>>>>>> > Ashish
>>> >>>>>>> >
>>> >>>>>>> >
>>> >>>>>>> >
>>> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>>> >>>>>>> >wrote:
>>> >>>>>>> >
>>> >>>>>>> >>
>>> >>>>>>> >> Hi,
>>> >>>>>>> >>
>>> >>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>>> >>>>>>> to
>>> >>>>>>> >> debug.
>>> >>>>>>> >>
>>> >>>>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
>>> >>>>>>> can
>>> >>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>>> >>>>>>> be
>>> >>>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>>> >>>>>>> >> server), but where do I specify that in the code in eclipse ?
>>> >>>>>>> >>
>>> >>>>>>> >> My next question is - what is the difference between MiniBSPCluster and
>>> >>>>>>> >> LocalBSPRunner  and which file should I be using to be able to step
>>> >>>>>>> through
>>> >>>>>>> >> the code ?
>>> >>>>>>> >>
>>> >>>>>>> >> These are the processes currently running on my machine -
>>> >>>>>>> >>
>>> >>>>>>> >> 6112 NameNode
>>> >>>>>>> >> 3319 ZooKeeperRunner
>>> >>>>>>> >> 3018
>>> >>>>>>> >> 6449 SecondaryNameNode
>>> >>>>>>> >> 5709 HMaster
>>> >>>>>>> >> 5643 HQuorumPeer
>>> >>>>>>> >> 6684 TaskTracker
>>> >>>>>>> >> 6526 JobTracker
>>> >>>>>>> >> 8074 Jps
>>> >>>>>>> >> 6277 DataNode
>>> >>>>>>> >>
>>> >>>>>>> >> Thank you,
>>> >>>>>>> >> Ashish
>>> >>>>>>> >>
>>> >>>>>>> >
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> --
>>> >>>>>>> Best Regards, Edward J. Yoon
>>> >>>>>>> @eddieyoon
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Best Regards, Edward J. Yoon
>>> >>> @eddieyoon
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Best Regards, Edward J. Yoon
>>> >> @eddieyoon
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Best Regards, Edward J. Yoon
>>> > @eddieyoon
>>> >
>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>

Re: Debugging hama in eclipse

Posted by "Edward J. Yoon" <ed...@apache.org>.

If you want to execute a command in the background without watching
logs, you can do the following:

% bin/hama zookeeper &

On Thu, Jun 23, 2011 at 4:43 PM, Ashish Agarwal <as...@gmail.com> wrote:
> How long does it take to execute the command
> bin/hama zookeeper
>
> I had to stop the execution as it stopped doing anything after some time.
>
> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper
> 11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
> GMT
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> environment:host.name=ubuntu.ubuntu-domain
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> environment:java.version=1.6.0_24
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> environment:java.vendor=Sun Microsystems Inc.
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> environment:java.io.tmpdir=/tmp
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> environment:java.compiler=<NA>
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.name=Linux
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.arch=i386
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> environment:os.version=2.6.35-28-generic
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> environment:user.name=hadoop
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> environment:user.home=/home/hadoop
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
> environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1
> 11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to 1200000
> 11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port
> 0.0.0.0/0.0.0.0:21810
> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
> 11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0
> ^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>
>
> Thanks
> Ashish
>
>
>
> On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd
> <jo...@cbsinteractive.com> wrote:
>>
>> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <ed...@apache.org> wrote:
>> > Instead removing 206 line, why don't we add dependencies as below?
>>
>> I think removing line 206 is the right thing to do.  Hama doesn't
>> depend on commons-cli, hadoop-core does, and maven will properly pick
>> that up and include the jar if allowed.
>>
>> The fact that Hama doesn't depend on the several items below was
>> correctly detected by maven, in the report description of HAMA-402.
>>
>> You don't mention a particular problem with including the transitive
>> dependencies, but if your problem is that it includes too many items,
>> maven does have options to exclude a particular transitive dependency
>> on a case-by-case basis that might be more appropriate to use.
>>
>> Including the dependencies below as you suggest puts Hama in the place
>> of maintaining a transitive dependency like commons-cli in lock-step
>> hadoop-core, where the actual dependency comes from.
>>
>>
>> This is all just my $.02 worth, I'm open to arguments in favor of
>> <excludeTransitive/>, and there is often more than one way to skin a
>> cat with respect to dependency management.
>>
>>
>> ...joe
>>
>>
>>
>> >    <dependency>
>> >      <groupId>org.mortbay.jetty</groupId>
>> >      <artifactId>jetty-annotations</artifactId>
>> >      <version>6.1H.22</version>
>> >    </dependency>
>> >    <dependency>
>> >      <groupId>org.slf4j</groupId>
>> >      <artifactId>slf4j-log4j12</artifactId>
>> >      <version>1.5.2</version>
>> >    </dependency>
>> >    <dependency>
>> >      <groupId>commons-cli</groupId>
>> >      <artifactId>commons-cli</artifactId>
>> >      <version>1.2</version>
>> >    </dependency>
>> >
>> > Thanks.
>> >
>> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <ed...@apache.org> wrote:
>> >> Oh... sorry, you're right.
>> >>
>> >> It looks like a bug. We have to fix this.
>> >>
>> >> [INFO] Total time: 4:36.098s
>> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
>> >> [INFO] Final Memory: 20M/51M
>> >> [INFO] ------------------------------------------------------------------------
>> >> edward@test ~/workspace/hama-trunk $ ls lib/
>> >> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>> >> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>> >> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>> >>    zookeeper-3.3.1.jar
>> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
>> >> -bash: bin/hama: Permission denied
>> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
>> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>> >> org/apache/commons/cli/ParseException
>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>> >>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>> >> Caused by: java.lang.ClassNotFoundException:
>> >> org.apache.commons.cli.ParseException
>> >>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> >>        at java.security.AccessController.doPrivileged(Native Method)
>> >>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
>> >>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> >>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
>> >>        ... 3 more
>> >>
>> >>
>> >>
>> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <ed...@apache.org> wrote:
>> >>> That's so weird.
>> >>>
>> >>> It works well on my test cluster.
>> >>>
>> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
>> >>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>> >>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>> >>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>> >>>    zookeeper-3.3.1.jar
>> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
>> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
>> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
>> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
>> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
>> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
>> >>> Estimated value of PI is 3.1534
>> >>> Job Finished in 7.886 seconds
>> >>> edward@slave:~/workspace/hama-trunk$
>> >>>
>> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
>> >>> <jo...@cbsinteractive.com> wrote:
>> >>>> Replying to my own post ...
>> >>>>
>> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>> >>>> <jo...@cbsinteractive.com> wrote:
>> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>> >>>>>>
>> >>>>>> When I try to run the hama daemons using the command -
>> >>>>>> bin/hama zookeeper
>> >>>>>>
>> >>>>>> I get this error -
>> >>>>
>> >>>> It looks like the problem is actually this line in pom.xml (line 205)  :
>> >>>>                    <excludeTransitive>true</excludeTransitive>
>> >>>>
>> >>>> This prevents commons-cli (and other transitive dependencies) from
>> >>>> being copied to the lib dir like they should be.
>> >>>>
>> >>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
>> >>>> JIRA for this after lunch, the command line is fairly broken with this
>> >>>> line in, especially after several dependencies were (correctly)
>> >>>> removed in HAMA-402.
>> >>>>
>> >>>>
>> >>>> ...joe
>> >>>>
>> >>>>
>> >>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
>> >>>>> commons-cli dependency was removed recently as part of HAMA-402, and
>> >>>>> re-adding it fixes the NoClassDefFoundError.
>> >>>>>
>> >>>>> I believe commons-cli should have been pulled as part of the
>> >>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
>> >>>>> rusty, and I'm no longer sure just how transitive dependencies are
>> >>>>> pulled)
>> >>>>>
>> >>>>> While a quick work-around is to re-add the dependency here, the proper
>> >>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
>> >>>>> depends on commons-cli.
>> >>>>>
>> >>>>>
>> >>>>> ...joe
>> >>>>>
>> >>>>>
>> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> >>>>>> org/apache/commons/cli/ParseException
>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>> >>>>>> Caused by: java.lang.ClassNotFoundException:
>> >>>>>> org.apache.commons.cli.ParseException
>> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> >>>>>> at java.security.AccessController.doPrivileged(Native Method)
>> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> >>>>>> ... 3 more
>> >>>>>>
>> >>>>>>
>> >>>>>> I tried installing it a few times, but can't think of a solution to the
>> >>>>>> problem. Can you please help ?
>> >>>>>>
>> >>>>>> Thank you
>> >>>>>>
>> >>>>>> Sincerely,
>> >>>>>> Ashish
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>> >>>>>>
>> >>>>>>> Hi,
>> >>>>>>>
>> >>>>>>> I would suggest following steps to debug:
>> >>>>>>>
>> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>> >>>>>>> 2. Run the Hama deamons with 3 terminals.
>> >>>>>>>
>> >>>>>>> % bin/hama zookeeper
>> >>>>>>> % bin/hama bspmaster
>> >>>>>>> % bin/hama groom
>> >>>>>>>
>> >>>>>>> 3. Use the logging API to log debug.
>> >>>>>>> For example, LOG.debug("like this");
>> >>>>>>>
>> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>> > Hi,
>> >>>>>>> >
>> >>>>>>> > I was wondering if anyone can give me instructions on how to debug code
>> >>>>>>> in
>> >>>>>>> > distributed or pseudo-distributed mode ? It would really help me.
>> >>>>>>> >
>> >>>>>>> > Thank you
>> >>>>>>> >
>> >>>>>>> > Sincerely,
>> >>>>>>> > Ashish
>> >>>>>>> >
>> >>>>>>> >
>> >>>>>>> >
>> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>> >>>>>>> >wrote:
>> >>>>>>> >
>> >>>>>>> >>
>> >>>>>>> >> Hi,
>> >>>>>>> >>
>> >>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>> >>>>>>> to
>> >>>>>>> >> debug.
>> >>>>>>> >>
>> >>>>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
>> >>>>>>> can
>> >>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>> >>>>>>> be
>> >>>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>> >>>>>>> >> server), but where do I specify that in the code in eclipse ?
>> >>>>>>> >>
>> >>>>>>> >> My next question is - what is the difference between MiniBSPCluster and
>> >>>>>>> >> LocalBSPRunner  and which file should I be using to be able to step
>> >>>>>>> through
>> >>>>>>> >> the code ?
>> >>>>>>> >>
>> >>>>>>> >> These are the processes currently running on my machine -
>> >>>>>>> >>
>> >>>>>>> >> 6112 NameNode
>> >>>>>>> >> 3319 ZooKeeperRunner
>> >>>>>>> >> 3018
>> >>>>>>> >> 6449 SecondaryNameNode
>> >>>>>>> >> 5709 HMaster
>> >>>>>>> >> 5643 HQuorumPeer
>> >>>>>>> >> 6684 TaskTracker
>> >>>>>>> >> 6526 JobTracker
>> >>>>>>> >> 8074 Jps
>> >>>>>>> >> 6277 DataNode
>> >>>>>>> >>
>> >>>>>>> >> Thank you,
>> >>>>>>> >> Ashish
>> >>>>>>> >>
>> >>>>>>> >
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>> Best Regards, Edward J. Yoon
>> >>>>>>> @eddieyoon
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Best Regards, Edward J. Yoon
>> >>> @eddieyoon
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards, Edward J. Yoon
>> >> @eddieyoon
>> >>
>> >
>> >
>> >
>> > --
>> > Best Regards, Edward J. Yoon
>> > @eddieyoon
>> >
>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: Debugging hama in eclipse

Posted by Ashish Agarwal <as...@gmail.com>.

How long does it take to execute the command
bin/hama zookeeper

I had to stop the execution as it stopped doing anything after some time.

hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper
11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
GMT
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:host.name=ubuntu.ubuntu-domain
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:java.version=1.6.0_24
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:java.vendor=Sun Microsystems Inc.
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:java.io.tmpdir=/tmp
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:java.compiler=<NA>
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.name=Linux
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.arch=i386
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:os.version=2.6.35-28-generic
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:user.name=hadoop
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:user.home=/home/hadoop
11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk
11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000
11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1
11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to 1200000
11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port
0.0.0.0/0.0.0.0:21810
11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
/tmp/hama-hadoop/zookeeper/version-2/snapshot.0
11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
/tmp/hama-hadoop/zookeeper/version-2/snapshot.0
11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0
^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$


Thanks
Ashish



On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd
<jo...@cbsinteractive.com> wrote:
>
> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <ed...@apache.org> wrote:
> > Instead removing 206 line, why don't we add dependencies as below?
>
> I think removing line 206 is the right thing to do.  Hama doesn't
> depend on commons-cli, hadoop-core does, and maven will properly pick
> that up and include the jar if allowed.
>
> The fact that Hama doesn't depend on the several items below was
> correctly detected by maven, in the report description of HAMA-402.
>
> You don't mention a particular problem with including the transitive
> dependencies, but if your problem is that it includes too many items,
> maven does have options to exclude a particular transitive dependency
> on a case-by-case basis that might be more appropriate to use.
>
> Including the dependencies below as you suggest puts Hama in the place
> of maintaining a transitive dependency like commons-cli in lock-step
> hadoop-core, where the actual dependency comes from.
>
>
> This is all just my $.02 worth, I'm open to arguments in favor of
> <excludeTransitive/>, and there is often more than one way to skin a
> cat with respect to dependency management.
>
>
> ...joe
>
>
>
> >    <dependency>
> >      <groupId>org.mortbay.jetty</groupId>
> >      <artifactId>jetty-annotations</artifactId>
> >      <version>6.1H.22</version>
> >    </dependency>
> >    <dependency>
> >      <groupId>org.slf4j</groupId>
> >      <artifactId>slf4j-log4j12</artifactId>
> >      <version>1.5.2</version>
> >    </dependency>
> >    <dependency>
> >      <groupId>commons-cli</groupId>
> >      <artifactId>commons-cli</artifactId>
> >      <version>1.2</version>
> >    </dependency>
> >
> > Thanks.
> >
> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <ed...@apache.org> wrote:
> >> Oh... sorry, you're right.
> >>
> >> It looks like a bug. We have to fix this.
> >>
> >> [INFO] Total time: 4:36.098s
> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
> >> [INFO] Final Memory: 20M/51M
> >> [INFO] ------------------------------------------------------------------------
> >> edward@test ~/workspace/hama-trunk $ ls lib/
> >> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
> >> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
> >> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
> >>    zookeeper-3.3.1.jar
> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
> >> -bash: bin/hama: Permission denied
> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
> >> Exception in thread "main" java.lang.NoClassDefFoundError:
> >> org/apache/commons/cli/ParseException
> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
> >>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> >>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
> >> Caused by: java.lang.ClassNotFoundException:
> >> org.apache.commons.cli.ParseException
> >>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> >>        at java.security.AccessController.doPrivileged(Native Method)
> >>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
> >>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
> >>        ... 3 more
> >>
> >>
> >>
> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <ed...@apache.org> wrote:
> >>> That's so weird.
> >>>
> >>> It works well on my test cluster.
> >>>
> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
> >>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
> >>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
> >>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
> >>>    zookeeper-3.3.1.jar
> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
> >>> Estimated value of PI is 3.1534
> >>> Job Finished in 7.886 seconds
> >>> edward@slave:~/workspace/hama-trunk$
> >>>
> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
> >>> <jo...@cbsinteractive.com> wrote:
> >>>> Replying to my own post ...
> >>>>
> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
> >>>> <jo...@cbsinteractive.com> wrote:
> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
> >>>>>>
> >>>>>> When I try to run the hama daemons using the command -
> >>>>>> bin/hama zookeeper
> >>>>>>
> >>>>>> I get this error -
> >>>>
> >>>> It looks like the problem is actually this line in pom.xml (line 205)  :
> >>>>                    <excludeTransitive>true</excludeTransitive>
> >>>>
> >>>> This prevents commons-cli (and other transitive dependencies) from
> >>>> being copied to the lib dir like they should be.
> >>>>
> >>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
> >>>> JIRA for this after lunch, the command line is fairly broken with this
> >>>> line in, especially after several dependencies were (correctly)
> >>>> removed in HAMA-402.
> >>>>
> >>>>
> >>>> ...joe
> >>>>
> >>>>
> >>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
> >>>>> commons-cli dependency was removed recently as part of HAMA-402, and
> >>>>> re-adding it fixes the NoClassDefFoundError.
> >>>>>
> >>>>> I believe commons-cli should have been pulled as part of the
> >>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
> >>>>> rusty, and I'm no longer sure just how transitive dependencies are
> >>>>> pulled)
> >>>>>
> >>>>> While a quick work-around is to re-add the dependency here, the proper
> >>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
> >>>>> depends on commons-cli.
> >>>>>
> >>>>>
> >>>>> ...joe
> >>>>>
> >>>>>
> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
> >>>>>> org/apache/commons/cli/ParseException
> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
> >>>>>> Caused by: java.lang.ClassNotFoundException:
> >>>>>> org.apache.commons.cli.ParseException
> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>>>>> at java.security.AccessController.doPrivileged(Native Method)
> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >>>>>> ... 3 more
> >>>>>>
> >>>>>>
> >>>>>> I tried installing it a few times, but can't think of a solution to the
> >>>>>> problem. Can you please help ?
> >>>>>>
> >>>>>> Thank you
> >>>>>>
> >>>>>> Sincerely,
> >>>>>> Ashish
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I would suggest following steps to debug:
> >>>>>>>
> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
> >>>>>>> 2. Run the Hama deamons with 3 terminals.
> >>>>>>>
> >>>>>>> % bin/hama zookeeper
> >>>>>>> % bin/hama bspmaster
> >>>>>>> % bin/hama groom
> >>>>>>>
> >>>>>>> 3. Use the logging API to log debug.
> >>>>>>> For example, LOG.debug("like this");
> >>>>>>>
> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
> >>>>>>> wrote:
> >>>>>>> > Hi,
> >>>>>>> >
> >>>>>>> > I was wondering if anyone can give me instructions on how to debug code
> >>>>>>> in
> >>>>>>> > distributed or pseudo-distributed mode ? It would really help me.
> >>>>>>> >
> >>>>>>> > Thank you
> >>>>>>> >
> >>>>>>> > Sincerely,
> >>>>>>> > Ashish
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
> >>>>>>> >wrote:
> >>>>>>> >
> >>>>>>> >>
> >>>>>>> >> Hi,
> >>>>>>> >>
> >>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
> >>>>>>> to
> >>>>>>> >> debug.
> >>>>>>> >>
> >>>>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
> >>>>>>> can
> >>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
> >>>>>>> be
> >>>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
> >>>>>>> >> server), but where do I specify that in the code in eclipse ?
> >>>>>>> >>
> >>>>>>> >> My next question is - what is the difference between MiniBSPCluster and
> >>>>>>> >> LocalBSPRunner  and which file should I be using to be able to step
> >>>>>>> through
> >>>>>>> >> the code ?
> >>>>>>> >>
> >>>>>>> >> These are the processes currently running on my machine -
> >>>>>>> >>
> >>>>>>> >> 6112 NameNode
> >>>>>>> >> 3319 ZooKeeperRunner
> >>>>>>> >> 3018
> >>>>>>> >> 6449 SecondaryNameNode
> >>>>>>> >> 5709 HMaster
> >>>>>>> >> 5643 HQuorumPeer
> >>>>>>> >> 6684 TaskTracker
> >>>>>>> >> 6526 JobTracker
> >>>>>>> >> 8074 Jps
> >>>>>>> >> 6277 DataNode
> >>>>>>> >>
> >>>>>>> >> Thank you,
> >>>>>>> >> Ashish
> >>>>>>> >>
> >>>>>>> >
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Best Regards, Edward J. Yoon
> >>>>>>> @eddieyoon
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards, Edward J. Yoon
> >>> @eddieyoon
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> @eddieyoon
> >>
> >
> >
> >
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
> >

Re: Debugging hama in eclipse

Posted by Joseph Boyd <jo...@cbsinteractive.com>.

On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <ed...@apache.org> wrote:
> Instead removing 206 line, why don't we add dependencies as below?

I think removing line 206 is the right thing to do.  Hama doesn't
depend on commons-cli, hadoop-core does, and maven will properly pick
that up and include the jar if allowed.

The fact that Hama doesn't depend on the several items below was
correctly detected by maven, in the report description of HAMA-402.

You don't mention a particular problem with including the transitive
dependencies, but if your problem is that it includes too many items,
maven does have options to exclude a particular transitive dependency
on a case-by-case basis that might be more appropriate to use.

Including the dependencies below as you suggest puts Hama in the place
of maintaining a transitive dependency like commons-cli in lock-step
hadoop-core, where the actual dependency comes from.


This is all just my $.02 worth, I'm open to arguments in favor of
<excludeTransitive/>, and there is often more than one way to skin a
cat with respect to dependency management.


...joe



>    <dependency>
>      <groupId>org.mortbay.jetty</groupId>
>      <artifactId>jetty-annotations</artifactId>
>      <version>6.1H.22</version>
>    </dependency>
>    <dependency>
>      <groupId>org.slf4j</groupId>
>      <artifactId>slf4j-log4j12</artifactId>
>      <version>1.5.2</version>
>    </dependency>
>    <dependency>
>      <groupId>commons-cli</groupId>
>      <artifactId>commons-cli</artifactId>
>      <version>1.2</version>
>    </dependency>
>
> Thanks.
>
> On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <ed...@apache.org> wrote:
>> Oh... sorry, you're right.
>>
>> It looks like a bug. We have to fix this.
>>
>> [INFO] Total time: 4:36.098s
>> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
>> [INFO] Final Memory: 20M/51M
>> [INFO] ------------------------------------------------------------------------
>> edward@test ~/workspace/hama-trunk $ ls lib/
>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>    zookeeper-3.3.1.jar
>> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
>> -bash: bin/hama: Permission denied
>> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
>> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/commons/cli/ParseException
>>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.commons.cli.ParseException
>>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
>>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
>>        ... 3 more
>>
>>
>>
>> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>> That's so weird.
>>>
>>> It works well on my test cluster.
>>>
>>> edward@slave:~/workspace/hama-trunk$ ls lib/
>>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>>    zookeeper-3.3.1.jar
>>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
>>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
>>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
>>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
>>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
>>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
>>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
>>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
>>> Estimated value of PI is 3.1534
>>> Job Finished in 7.886 seconds
>>> edward@slave:~/workspace/hama-trunk$
>>>
>>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
>>> <jo...@cbsinteractive.com> wrote:
>>>> Replying to my own post ...
>>>>
>>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>>>> <jo...@cbsinteractive.com> wrote:
>>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>>>>>>
>>>>>> When I try to run the hama daemons using the command -
>>>>>> bin/hama zookeeper
>>>>>>
>>>>>> I get this error -
>>>>
>>>> It looks like the problem is actually this line in pom.xml (line 205)  :
>>>>                    <excludeTransitive>true</excludeTransitive>
>>>>
>>>> This prevents commons-cli (and other transitive dependencies) from
>>>> being copied to the lib dir like they should be.
>>>>
>>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
>>>> JIRA for this after lunch, the command line is fairly broken with this
>>>> line in, especially after several dependencies were (correctly)
>>>> removed in HAMA-402.
>>>>
>>>>
>>>> ...joe
>>>>
>>>>
>>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
>>>>> commons-cli dependency was removed recently as part of HAMA-402, and
>>>>> re-adding it fixes the NoClassDefFoundError.
>>>>>
>>>>> I believe commons-cli should have been pulled as part of the
>>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
>>>>> rusty, and I'm no longer sure just how transitive dependencies are
>>>>> pulled)
>>>>>
>>>>> While a quick work-around is to re-add the dependency here, the proper
>>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
>>>>> depends on commons-cli.
>>>>>
>>>>>
>>>>> ...joe
>>>>>
>>>>>
>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>> org/apache/commons/cli/ParseException
>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>> org.apache.commons.cli.ParseException
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>>>> ... 3 more
>>>>>>
>>>>>>
>>>>>> I tried installing it a few times, but can't think of a solution to the
>>>>>> problem. Can you please help ?
>>>>>>
>>>>>> Thank you
>>>>>>
>>>>>> Sincerely,
>>>>>> Ashish
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I would suggest following steps to debug:
>>>>>>>
>>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>>>>>>> 2. Run the Hama deamons with 3 terminals.
>>>>>>>
>>>>>>> % bin/hama zookeeper
>>>>>>> % bin/hama bspmaster
>>>>>>> % bin/hama groom
>>>>>>>
>>>>>>> 3. Use the logging API to log debug.
>>>>>>> For example, LOG.debug("like this");
>>>>>>>
>>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>>>>>>> wrote:
>>>>>>> > Hi,
>>>>>>> >
>>>>>>> > I was wondering if anyone can give me instructions on how to debug code
>>>>>>> in
>>>>>>> > distributed or pseudo-distributed mode ? It would really help me.
>>>>>>> >
>>>>>>> > Thank you
>>>>>>> >
>>>>>>> > Sincerely,
>>>>>>> > Ashish
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>>>>>>> >wrote:
>>>>>>> >
>>>>>>> >>
>>>>>>> >> Hi,
>>>>>>> >>
>>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>>>>>>> to
>>>>>>> >> debug.
>>>>>>> >>
>>>>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
>>>>>>> can
>>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>>>>>>> be
>>>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>>>>>>> >> server), but where do I specify that in the code in eclipse ?
>>>>>>> >>
>>>>>>> >> My next question is - what is the difference between MiniBSPCluster and
>>>>>>> >> LocalBSPRunner  and which file should I be using to be able to step
>>>>>>> through
>>>>>>> >> the code ?
>>>>>>> >>
>>>>>>> >> These are the processes currently running on my machine -
>>>>>>> >>
>>>>>>> >> 6112 NameNode
>>>>>>> >> 3319 ZooKeeperRunner
>>>>>>> >> 3018
>>>>>>> >> 6449 SecondaryNameNode
>>>>>>> >> 5709 HMaster
>>>>>>> >> 5643 HQuorumPeer
>>>>>>> >> 6684 TaskTracker
>>>>>>> >> 6526 JobTracker
>>>>>>> >> 8074 Jps
>>>>>>> >> 6277 DataNode
>>>>>>> >>
>>>>>>> >> Thank you,
>>>>>>> >> Ashish
>>>>>>> >>
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards, Edward J. Yoon
>>>>>>> @eddieyoon
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>

Re: Debugging hama in eclipse

Posted by "Edward J. Yoon" <ed...@apache.org>.

I found another problem about ClassNotFoundException.

java.lang.ClassNotFoundException: org.mortbay.jetty.annotations.Configuration

> It looks like the problem is actually this line in pom.xml (line 205)  :
>                   <excludeTransitive>true</excludeTransitive>

Instead removing 206 line, why don't we add dependencies as below?

    <dependency>
      <groupId>org.mortbay.jetty</groupId>
      <artifactId>jetty-annotations</artifactId>
      <version>6.1H.22</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-log4j12</artifactId>
      <version>1.5.2</version>
    </dependency>
    <dependency>
      <groupId>commons-cli</groupId>
      <artifactId>commons-cli</artifactId>
      <version>1.2</version>
    </dependency>

Thanks.

On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <ed...@apache.org> wrote:
> Oh... sorry, you're right.
>
> It looks like a bug. We have to fix this.
>
> [INFO] Total time: 4:36.098s
> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
> [INFO] Final Memory: 20M/51M
> [INFO] ------------------------------------------------------------------------
> edward@test ~/workspace/hama-trunk $ ls lib/
> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>    zookeeper-3.3.1.jar
> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
> -bash: bin/hama: Permission denied
> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/commons/cli/ParseException
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>        at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.commons.cli.ParseException
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
>        ... 3 more
>
>
>
> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <ed...@apache.org> wrote:
>> That's so weird.
>>
>> It works well on my test cluster.
>>
>> edward@slave:~/workspace/hama-trunk$ ls lib/
>> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
>> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
>> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>>    zookeeper-3.3.1.jar
>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
>> Estimated value of PI is 3.1534
>> Job Finished in 7.886 seconds
>> edward@slave:~/workspace/hama-trunk$
>>
>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
>> <jo...@cbsinteractive.com> wrote:
>>> Replying to my own post ...
>>>
>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>>> <jo...@cbsinteractive.com> wrote:
>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>>>>>
>>>>> When I try to run the hama daemons using the command -
>>>>> bin/hama zookeeper
>>>>>
>>>>> I get this error -
>>>
>>> It looks like the problem is actually this line in pom.xml (line 205)  :
>>>                    <excludeTransitive>true</excludeTransitive>
>>>
>>> This prevents commons-cli (and other transitive dependencies) from
>>> being copied to the lib dir like they should be.
>>>
>>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
>>> JIRA for this after lunch, the command line is fairly broken with this
>>> line in, especially after several dependencies were (correctly)
>>> removed in HAMA-402.
>>>
>>>
>>> ...joe
>>>
>>>
>>>> I'm getting the same NoClassDefFoundError using the svn code.  The
>>>> commons-cli dependency was removed recently as part of HAMA-402, and
>>>> re-adding it fixes the NoClassDefFoundError.
>>>>
>>>> I believe commons-cli should have been pulled as part of the
>>>> hadoop-core dependency, and wasn't.  (though my maven skills are
>>>> rusty, and I'm no longer sure just how transitive dependencies are
>>>> pulled)
>>>>
>>>> While a quick work-around is to re-add the dependency here, the proper
>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
>>>> depends on commons-cli.
>>>>
>>>>
>>>> ...joe
>>>>
>>>>
>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>> org/apache/commons/cli/ParseException
>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>> org.apache.commons.cli.ParseException
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>>> ... 3 more
>>>>>
>>>>>
>>>>> I tried installing it a few times, but can't think of a solution to the
>>>>> problem. Can you please help ?
>>>>>
>>>>> Thank you
>>>>>
>>>>> Sincerely,
>>>>> Ashish
>>>>>
>>>>>
>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I would suggest following steps to debug:
>>>>>>
>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>>>>>> 2. Run the Hama deamons with 3 terminals.
>>>>>>
>>>>>> % bin/hama zookeeper
>>>>>> % bin/hama bspmaster
>>>>>> % bin/hama groom
>>>>>>
>>>>>> 3. Use the logging API to log debug.
>>>>>> For example, LOG.debug("like this");
>>>>>>
>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>>>>>> wrote:
>>>>>> > Hi,
>>>>>> >
>>>>>> > I was wondering if anyone can give me instructions on how to debug code
>>>>>> in
>>>>>> > distributed or pseudo-distributed mode ? It would really help me.
>>>>>> >
>>>>>> > Thank you
>>>>>> >
>>>>>> > Sincerely,
>>>>>> > Ashish
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>>>>>> >wrote:
>>>>>> >
>>>>>> >>
>>>>>> >> Hi,
>>>>>> >>
>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>>>>>> to
>>>>>> >> debug.
>>>>>> >>
>>>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
>>>>>> can
>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>>>>>> be
>>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>>>>>> >> server), but where do I specify that in the code in eclipse ?
>>>>>> >>
>>>>>> >> My next question is - what is the difference between MiniBSPCluster and
>>>>>> >> LocalBSPRunner  and which file should I be using to be able to step
>>>>>> through
>>>>>> >> the code ?
>>>>>> >>
>>>>>> >> These are the processes currently running on my machine -
>>>>>> >>
>>>>>> >> 6112 NameNode
>>>>>> >> 3319 ZooKeeperRunner
>>>>>> >> 3018
>>>>>> >> 6449 SecondaryNameNode
>>>>>> >> 5709 HMaster
>>>>>> >> 5643 HQuorumPeer
>>>>>> >> 6684 TaskTracker
>>>>>> >> 6526 JobTracker
>>>>>> >> 8074 Jps
>>>>>> >> 6277 DataNode
>>>>>> >>
>>>>>> >> Thank you,
>>>>>> >> Ashish
>>>>>> >>
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards, Edward J. Yoon
>>>>>> @eddieyoon
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: Debugging hama in eclipse

Posted by "Edward J. Yoon" <ed...@apache.org>.

Oh... sorry, you're right.

It looks like a bug. We have to fix this.

[INFO] Total time: 4:36.098s
[INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
[INFO] Final Memory: 20M/51M
[INFO] ------------------------------------------------------------------------
edward@test ~/workspace/hama-trunk $ ls lib/
commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
jetty-util-6.1H.22.jar  log4j-1.2.15.jar
hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
    zookeeper-3.3.1.jar
edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
-bash: bin/hama: Permission denied
edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/commons/cli/ParseException
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.cli.ParseException
	at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
	... 3 more



On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <ed...@apache.org> wrote:
> That's so weird.
>
> It works well on my test cluster.
>
> edward@slave:~/workspace/hama-trunk$ ls lib/
> commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
> jetty-util-6.1H.22.jar  log4j-1.2.15.jar
> hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
>    zookeeper-3.3.1.jar
> edward@slave:~/workspace/hama-trunk$ bin/hama jar
> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
> Estimated value of PI is 3.1534
> Job Finished in 7.886 seconds
> edward@slave:~/workspace/hama-trunk$
>
> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
> <jo...@cbsinteractive.com> wrote:
>> Replying to my own post ...
>>
>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>> <jo...@cbsinteractive.com> wrote:
>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>>>>
>>>> When I try to run the hama daemons using the command -
>>>> bin/hama zookeeper
>>>>
>>>> I get this error -
>>
>> It looks like the problem is actually this line in pom.xml (line 205)  :
>>                    <excludeTransitive>true</excludeTransitive>
>>
>> This prevents commons-cli (and other transitive dependencies) from
>> being copied to the lib dir like they should be.
>>
>> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
>> JIRA for this after lunch, the command line is fairly broken with this
>> line in, especially after several dependencies were (correctly)
>> removed in HAMA-402.
>>
>>
>> ...joe
>>
>>
>>> I'm getting the same NoClassDefFoundError using the svn code.  The
>>> commons-cli dependency was removed recently as part of HAMA-402, and
>>> re-adding it fixes the NoClassDefFoundError.
>>>
>>> I believe commons-cli should have been pulled as part of the
>>> hadoop-core dependency, and wasn't.  (though my maven skills are
>>> rusty, and I'm no longer sure just how transitive dependencies are
>>> pulled)
>>>
>>> While a quick work-around is to re-add the dependency here, the proper
>>> fix is probably to fix the hadoop-core pom, and properly mark that it
>>> depends on commons-cli.
>>>
>>>
>>> ...joe
>>>
>>>
>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> org/apache/commons/cli/ParseException
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> org.apache.commons.cli.ParseException
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>> ... 3 more
>>>>
>>>>
>>>> I tried installing it a few times, but can't think of a solution to the
>>>> problem. Can you please help ?
>>>>
>>>> Thank you
>>>>
>>>> Sincerely,
>>>> Ashish
>>>>
>>>>
>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I would suggest following steps to debug:
>>>>>
>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>>>>> 2. Run the Hama deamons with 3 terminals.
>>>>>
>>>>> % bin/hama zookeeper
>>>>> % bin/hama bspmaster
>>>>> % bin/hama groom
>>>>>
>>>>> 3. Use the logging API to log debug.
>>>>> For example, LOG.debug("like this");
>>>>>
>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>>>>> wrote:
>>>>> > Hi,
>>>>> >
>>>>> > I was wondering if anyone can give me instructions on how to debug code
>>>>> in
>>>>> > distributed or pseudo-distributed mode ? It would really help me.
>>>>> >
>>>>> > Thank you
>>>>> >
>>>>> > Sincerely,
>>>>> > Ashish
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>>>>> >wrote:
>>>>> >
>>>>> >>
>>>>> >> Hi,
>>>>> >>
>>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>>>>> to
>>>>> >> debug.
>>>>> >>
>>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
>>>>> can
>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>>>>> be
>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>>>>> >> server), but where do I specify that in the code in eclipse ?
>>>>> >>
>>>>> >> My next question is - what is the difference between MiniBSPCluster and
>>>>> >> LocalBSPRunner  and which file should I be using to be able to step
>>>>> through
>>>>> >> the code ?
>>>>> >>
>>>>> >> These are the processes currently running on my machine -
>>>>> >>
>>>>> >> 6112 NameNode
>>>>> >> 3319 ZooKeeperRunner
>>>>> >> 3018
>>>>> >> 6449 SecondaryNameNode
>>>>> >> 5709 HMaster
>>>>> >> 5643 HQuorumPeer
>>>>> >> 6684 TaskTracker
>>>>> >> 6526 JobTracker
>>>>> >> 8074 Jps
>>>>> >> 6277 DataNode
>>>>> >>
>>>>> >> Thank you,
>>>>> >> Ashish
>>>>> >>
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards, Edward J. Yoon
>>>>> @eddieyoon
>>>>>
>>>>
>>>
>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: Debugging hama in eclipse

Posted by "Edward J. Yoon" <ed...@apache.org>.

That's so weird.

It works well on my test cluster.

edward@slave:~/workspace/hama-trunk$ ls lib/
commons-logging-1.0.4.jar  hadoop-test-0.20.2.jar
jetty-util-6.1H.22.jar  log4j-1.2.15.jar
hadoop-core-0.20.2.jar     jetty-6.1H.22.jar       junit-3.8.1.jar
    zookeeper-3.3.1.jar
edward@slave:~/workspace/hama-trunk$ bin/hama jar
hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001
11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1
Estimated value of PI is 3.1534
Job Finished in 7.886 seconds
edward@slave:~/workspace/hama-trunk$

On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
<jo...@cbsinteractive.com> wrote:
> Replying to my own post ...
>
> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
> <jo...@cbsinteractive.com> wrote:
>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>>>
>>> When I try to run the hama daemons using the command -
>>> bin/hama zookeeper
>>>
>>> I get this error -
>
> It looks like the problem is actually this line in pom.xml (line 205)  :
>                    <excludeTransitive>true</excludeTransitive>
>
> This prevents commons-cli (and other transitive dependencies) from
> being copied to the lib dir like they should be.
>
> Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
> JIRA for this after lunch, the command line is fairly broken with this
> line in, especially after several dependencies were (correctly)
> removed in HAMA-402.
>
>
> ...joe
>
>
>> I'm getting the same NoClassDefFoundError using the svn code.  The
>> commons-cli dependency was removed recently as part of HAMA-402, and
>> re-adding it fixes the NoClassDefFoundError.
>>
>> I believe commons-cli should have been pulled as part of the
>> hadoop-core dependency, and wasn't.  (though my maven skills are
>> rusty, and I'm no longer sure just how transitive dependencies are
>> pulled)
>>
>> While a quick work-around is to re-add the dependency here, the proper
>> fix is probably to fix the hadoop-core pom, and properly mark that it
>> depends on commons-cli.
>>
>>
>> ...joe
>>
>>
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> org/apache/commons/cli/ParseException
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.commons.cli.ParseException
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> ... 3 more
>>>
>>>
>>> I tried installing it a few times, but can't think of a solution to the
>>> problem. Can you please help ?
>>>
>>> Thank you
>>>
>>> Sincerely,
>>> Ashish
>>>
>>>
>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>>>
>>>> Hi,
>>>>
>>>> I would suggest following steps to debug:
>>>>
>>>> 1. Install the Hadoop 0.20.x on your laptop.
>>>> 2. Run the Hama deamons with 3 terminals.
>>>>
>>>> % bin/hama zookeeper
>>>> % bin/hama bspmaster
>>>> % bin/hama groom
>>>>
>>>> 3. Use the logging API to log debug.
>>>> For example, LOG.debug("like this");
>>>>
>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>>>> wrote:
>>>> > Hi,
>>>> >
>>>> > I was wondering if anyone can give me instructions on how to debug code
>>>> in
>>>> > distributed or pseudo-distributed mode ? It would really help me.
>>>> >
>>>> > Thank you
>>>> >
>>>> > Sincerely,
>>>> > Ashish
>>>> >
>>>> >
>>>> >
>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>>>> >wrote:
>>>> >
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>>>> to
>>>> >> debug.
>>>> >>
>>>> >> I am trying to set up the environment into pseudo-distributed mode so I
>>>> can
>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>>>> be
>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>>>> >> server), but where do I specify that in the code in eclipse ?
>>>> >>
>>>> >> My next question is - what is the difference between MiniBSPCluster and
>>>> >> LocalBSPRunner  and which file should I be using to be able to step
>>>> through
>>>> >> the code ?
>>>> >>
>>>> >> These are the processes currently running on my machine -
>>>> >>
>>>> >> 6112 NameNode
>>>> >> 3319 ZooKeeperRunner
>>>> >> 3018
>>>> >> 6449 SecondaryNameNode
>>>> >> 5709 HMaster
>>>> >> 5643 HQuorumPeer
>>>> >> 6684 TaskTracker
>>>> >> 6526 JobTracker
>>>> >> 8074 Jps
>>>> >> 6277 DataNode
>>>> >>
>>>> >> Thank you,
>>>> >> Ashish
>>>> >>
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards, Edward J. Yoon
>>>> @eddieyoon
>>>>
>>>
>>
>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: Debugging hama in eclipse

Posted by Joseph Boyd <jo...@cbsinteractive.com>.

Replying to my own post ...

On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
<jo...@cbsinteractive.com> wrote:
> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>>
>> When I try to run the hama daemons using the command -
>> bin/hama zookeeper
>>
>> I get this error -

It looks like the problem is actually this line in pom.xml (line 205)  :
                    <excludeTransitive>true</excludeTransitive>

This prevents commons-cli (and other transitive dependencies) from
being copied to the lib dir like they should be.

Removing line 205 fixes the NoClassDefFoundError for me.  I can open a
JIRA for this after lunch, the command line is fairly broken with this
line in, especially after several dependencies were (correctly)
removed in HAMA-402.


...joe


> I'm getting the same NoClassDefFoundError using the svn code.  The
> commons-cli dependency was removed recently as part of HAMA-402, and
> re-adding it fixes the NoClassDefFoundError.
>
> I believe commons-cli should have been pulled as part of the
> hadoop-core dependency, and wasn't.  (though my maven skills are
> rusty, and I'm no longer sure just how transitive dependencies are
> pulled)
>
> While a quick work-around is to re-add the dependency here, the proper
> fix is probably to fix the hadoop-core pom, and properly mark that it
> depends on commons-cli.
>
>
> ...joe
>
>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/commons/cli/ParseException
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.commons.cli.ParseException
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> ... 3 more
>>
>>
>> I tried installing it a few times, but can't think of a solution to the
>> problem. Can you please help ?
>>
>> Thank you
>>
>> Sincerely,
>> Ashish
>>
>>
>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>>
>>> Hi,
>>>
>>> I would suggest following steps to debug:
>>>
>>> 1. Install the Hadoop 0.20.x on your laptop.
>>> 2. Run the Hama deamons with 3 terminals.
>>>
>>> % bin/hama zookeeper
>>> % bin/hama bspmaster
>>> % bin/hama groom
>>>
>>> 3. Use the logging API to log debug.
>>> For example, LOG.debug("like this");
>>>
>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>>> wrote:
>>> > Hi,
>>> >
>>> > I was wondering if anyone can give me instructions on how to debug code
>>> in
>>> > distributed or pseudo-distributed mode ? It would really help me.
>>> >
>>> > Thank you
>>> >
>>> > Sincerely,
>>> > Ashish
>>> >
>>> >
>>> >
>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>>> >wrote:
>>> >
>>> >>
>>> >> Hi,
>>> >>
>>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>>> to
>>> >> debug.
>>> >>
>>> >> I am trying to set up the environment into pseudo-distributed mode so I
>>> can
>>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>>> be
>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>>> >> server), but where do I specify that in the code in eclipse ?
>>> >>
>>> >> My next question is - what is the difference between MiniBSPCluster and
>>> >> LocalBSPRunner  and which file should I be using to be able to step
>>> through
>>> >> the code ?
>>> >>
>>> >> These are the processes currently running on my machine -
>>> >>
>>> >> 6112 NameNode
>>> >> 3319 ZooKeeperRunner
>>> >> 3018
>>> >> 6449 SecondaryNameNode
>>> >> 5709 HMaster
>>> >> 5643 HQuorumPeer
>>> >> 6684 TaskTracker
>>> >> 6526 JobTracker
>>> >> 8074 Jps
>>> >> 6277 DataNode
>>> >>
>>> >> Thank you,
>>> >> Ashish
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>>
>>
>

Re: Debugging hama in eclipse

Posted by Joseph Boyd <jo...@cbsinteractive.com>.

On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <as...@gmail.com> wrote:
>
> When I try to run the hama daemons using the command -
> bin/hama zookeeper
>
> I get this error -

I'm getting the same NoClassDefFoundError using the svn code.  The
commons-cli dependency was removed recently as part of HAMA-402, and
re-adding it fixes the NoClassDefFoundError.

I believe commons-cli should have been pulled as part of the
hadoop-core dependency, and wasn't.  (though my maven skills are
rusty, and I'm no longer sure just how transitive dependencies are
pulled)

While a quick work-around is to re-add the dependency here, the proper
fix is probably to fix the hadoop-core pom, and properly mark that it
depends on commons-cli.


...joe


> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/commons/cli/ParseException
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.commons.cli.ParseException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> ... 3 more
>
>
> I tried installing it a few times, but can't think of a solution to the
> problem. Can you please help ?
>
> Thank you
>
> Sincerely,
> Ashish
>
>
> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:
>
>> Hi,
>>
>> I would suggest following steps to debug:
>>
>> 1. Install the Hadoop 0.20.x on your laptop.
>> 2. Run the Hama deamons with 3 terminals.
>>
>> % bin/hama zookeeper
>> % bin/hama bspmaster
>> % bin/hama groom
>>
>> 3. Use the logging API to log debug.
>> For example, LOG.debug("like this");
>>
>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I was wondering if anyone can give me instructions on how to debug code
>> in
>> > distributed or pseudo-distributed mode ? It would really help me.
>> >
>> > Thank you
>> >
>> > Sincerely,
>> > Ashish
>> >
>> >
>> >
>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
>> >wrote:
>> >
>> >>
>> >> Hi,
>> >>
>> >> I have written some code for my GSoC task (Hama-367) and am now trying
>> to
>> >> debug.
>> >>
>> >> I am trying to set up the environment into pseudo-distributed mode so I
>> can
>> >> debug. I think the value of bsp.master.address in hama-site.xml has to
>> be
>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>> >> server), but where do I specify that in the code in eclipse ?
>> >>
>> >> My next question is - what is the difference between MiniBSPCluster and
>> >> LocalBSPRunner  and which file should I be using to be able to step
>> through
>> >> the code ?
>> >>
>> >> These are the processes currently running on my machine -
>> >>
>> >> 6112 NameNode
>> >> 3319 ZooKeeperRunner
>> >> 3018
>> >> 6449 SecondaryNameNode
>> >> 5709 HMaster
>> >> 5643 HQuorumPeer
>> >> 6684 TaskTracker
>> >> 6526 JobTracker
>> >> 8074 Jps
>> >> 6277 DataNode
>> >>
>> >> Thank you,
>> >> Ashish
>> >>
>> >
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>

Re: Debugging hama in eclipse

Posted by Thomas Jungblut <th...@googlemail.com>.

Just start using the bash scripts, I don't trust the CLI interfaces ;))

2011/6/16 Ashish Agarwal <as...@gmail.com>

> Hi,
>
> When I try to run the hama daemons using the command -
> bin/hama zookeeper
>
> I get this error -
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/commons/cli/ParseException
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.commons.cli.ParseException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> ... 3 more
>
>
> I tried installing it a few times, but can't think of a solution to the
> problem. Can you please help ?
>
> Thank you
>
> Sincerely,
> Ashish
>
>
> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <edwardyoon@apache.org
> >wrote:
>
> > Hi,
> >
> > I would suggest following steps to debug:
> >
> > 1. Install the Hadoop 0.20.x on your laptop.
> > 2. Run the Hama deamons with 3 terminals.
> >
> > % bin/hama zookeeper
> > % bin/hama bspmaster
> > % bin/hama groom
> >
> > 3. Use the logging API to log debug.
> > For example, LOG.debug("like this");
> >
> > On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <ashish.a77777@gmail.com
> >
> > wrote:
> > > Hi,
> > >
> > > I was wondering if anyone can give me instructions on how to debug code
> > in
> > > distributed or pseudo-distributed mode ? It would really help me.
> > >
> > > Thank you
> > >
> > > Sincerely,
> > > Ashish
> > >
> > >
> > >
> > > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <
> ashish.a77777@gmail.com
> > >wrote:
> > >
> > >>
> > >> Hi,
> > >>
> > >> I have written some code for my GSoC task (Hama-367) and am now trying
> > to
> > >> debug.
> > >>
> > >> I am trying to set up the environment into pseudo-distributed mode so
> I
> > can
> > >> debug. I think the value of bsp.master.address in hama-site.xml has to
> > be
> > >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
> > >> server), but where do I specify that in the code in eclipse ?
> > >>
> > >> My next question is - what is the difference between MiniBSPCluster
> and
> > >> LocalBSPRunner  and which file should I be using to be able to step
> > through
> > >> the code ?
> > >>
> > >> These are the processes currently running on my machine -
> > >>
> > >> 6112 NameNode
> > >> 3319 ZooKeeperRunner
> > >> 3018
> > >> 6449 SecondaryNameNode
> > >> 5709 HMaster
> > >> 5643 HQuorumPeer
> > >> 6684 TaskTracker
> > >> 6526 JobTracker
> > >> 8074 Jps
> > >> 6277 DataNode
> > >>
> > >> Thank you,
> > >> Ashish
> > >>
> > >
> >
> >
> >
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
> >
>



-- 
Thomas Jungblut
Berlin

mobile: 0170-3081070

business: thomas.jungblut@testberichte.de
private: thomas.jungblut@gmail.com

Re: Debugging hama in eclipse

Posted by Ashish Agarwal <as...@gmail.com>.

Hi,

When I try to run the hama daemons using the command -
bin/hama zookeeper

I get this error -

Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/commons/cli/ParseException
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.cli.ParseException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
... 3 more

I tried installing it a few times, but can't think of a solution to the
problem. Can you please help ?

Thank you

Sincerely,
Ashish

On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon <ed...@apache.org>wrote:

> Hi,
>
> I would suggest following steps to debug:
>
> 1. Install the Hadoop 0.20.x on your laptop.
> 2. Run the Hama deamons with 3 terminals.
>
> % bin/hama zookeeper
> % bin/hama bspmaster
> % bin/hama groom
>
> 3. Use the logging API to log debug.
> For example, LOG.debug("like this");
>
> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com>
> wrote:
> > Hi,
> >
> > I was wondering if anyone can give me instructions on how to debug code
> in
> > distributed or pseudo-distributed mode ? It would really help me.
> >
> > Thank you
> >
> > Sincerely,
> > Ashish
> >
> >
> >
> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <ashish.a77777@gmail.com
> >wrote:
> >
> >>
> >> Hi,
> >>
> >> I have written some code for my GSoC task (Hama-367) and am now trying
> to
> >> debug.
> >>
> >> I am trying to set up the environment into pseudo-distributed mode so I
> can
> >> debug. I think the value of bsp.master.address in hama-site.xml has to
> be
> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
> >> server), but where do I specify that in the code in eclipse ?
> >>
> >> My next question is - what is the difference between MiniBSPCluster and
> >> LocalBSPRunner  and which file should I be using to be able to step
> through
> >> the code ?
> >>
> >> These are the processes currently running on my machine -
> >>
> >> 6112 NameNode
> >> 3319 ZooKeeperRunner
> >> 3018
> >> 6449 SecondaryNameNode
> >> 5709 HMaster
> >> 5643 HQuorumPeer
> >> 6684 TaskTracker
> >> 6526 JobTracker
> >> 8074 Jps
> >> 6277 DataNode
> >>
> >> Thank you,
> >> Ashish
> >>
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>

Re: Debugging hama in eclipse

Posted by "Edward J. Yoon" <ed...@apache.org>.

Hi,

I would suggest following steps to debug:

1. Install the Hadoop 0.20.x on your laptop.
2. Run the Hama deamons with 3 terminals.

% bin/hama zookeeper
% bin/hama bspmaster
% bin/hama groom

3. Use the logging API to log debug.
For example, LOG.debug("like this");

On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal <as...@gmail.com> wrote:
> Hi,
>
> I was wondering if anyone can give me instructions on how to debug code in
> distributed or pseudo-distributed mode ? It would really help me.
>
> Thank you
>
> Sincerely,
> Ashish
>
>
>
> On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <as...@gmail.com>wrote:
>
>>
>> Hi,
>>
>> I have written some code for my GSoC task (Hama-367) and am now trying to
>> debug.
>>
>> I am trying to set up the environment into pseudo-distributed mode so I can
>> debug. I think the value of bsp.master.address in hama-site.xml has to be
>> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
>> server), but where do I specify that in the code in eclipse ?
>>
>> My next question is - what is the difference between MiniBSPCluster and
>> LocalBSPRunner  and which file should I be using to be able to step through
>> the code ?
>>
>> These are the processes currently running on my machine -
>>
>> 6112 NameNode
>> 3319 ZooKeeperRunner
>> 3018
>> 6449 SecondaryNameNode
>> 5709 HMaster
>> 5643 HQuorumPeer
>> 6684 TaskTracker
>> 6526 JobTracker
>> 8074 Jps
>> 6277 DataNode
>>
>> Thank you,
>> Ashish
>>
>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: Debugging hama in eclipse

Posted by Ashish Agarwal <as...@gmail.com>.

Hi,

I was wondering if anyone can give me instructions on how to debug code in
distributed or pseudo-distributed mode ? It would really help me.

Thank you

Sincerely,
Ashish



On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal <as...@gmail.com>wrote:

>
> Hi,
>
> I have written some code for my GSoC task (Hama-367) and am now trying to
> debug.
>
> I am trying to set up the environment into pseudo-distributed mode so I can
> debug. I think the value of bsp.master.address in hama-site.xml has to be
> local as this means LocalBSPRunner (as localhost would call 127.0.0.1
> server), but where do I specify that in the code in eclipse ?
>
> My next question is - what is the difference between MiniBSPCluster and
> LocalBSPRunner  and which file should I be using to be able to step through
> the code ?
>
> These are the processes currently running on my machine -
>
> 6112 NameNode
> 3319 ZooKeeperRunner
> 3018
> 6449 SecondaryNameNode
> 5709 HMaster
> 5643 HQuorumPeer
> 6684 TaskTracker
> 6526 JobTracker
> 8074 Jps
> 6277 DataNode
>
> Thank you,
> Ashish
>