You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Nick Klosterman <nk...@ecn.purdue.edu> on 2010/02/10 21:42:26 UTC
Ubuntu Single Node Tutorial failure. No live or dead nodes.
I've been following Michael Noll's Single node cluster tutorial but am
unable to run the wordcount example successfully.
It appears that I'm having some sort of problem involving the nodes. Using
copyFromLocal fails to replicate the data across 1 node.
When I try to look at the hadoop web interfaces I see that there aren't
any nodes (alive or dead) listed.
After I start things this is what I get from dfsadmin -report
/usr/local/hadoop/bin$ ./hadoop dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: %
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 0 (0 total, 0 dead)
Here are the commands I'm entering and the output of them:
/usr/local/hadoop/bin$ ./start-all.sh
starting namenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
localhost: starting datanode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
localhost: starting secondarynamenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
starting jobtracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
localhost: starting tasktracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
/usr/local/hadoop/bin$ jps
24440 SecondaryNameNode
24626 TaskTracker
24527 JobTracker
24218 NameNode
24725 Jps
---> I had all ready created the txtinput directory with ./hadoop dfs
-mkdir txtinput
/usr/local/hadoop/bin$ ./hadoop dfs -copyFromLocal
/home/hadoop/Desktop/*.txt txtinput
10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/hadoop/txtinput/20417.txt could only be replicated to 0 nodes,
instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
10/02/10 15:29:38 WARN hdfs.DFSClient: Error Recovery for block null bad
datanode[0] nodes == null
10/02/10 15:29:38 WARN hdfs.DFSClient: Could not get block locations.
Source file "/user/hadoop/txtinput/20417.txt" - Aborting...
10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/hadoop/txtinput/7ldvc10.txt could only be replicated to 0 nodes,
instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
10/02/10 15:29:38 WARN hdfs.DFSClient: Error Recovery for block null bad
datanode[0] nodes == null
10/02/10 15:29:38 WARN hdfs.DFSClient: Could not get block locations.
Source file "/user/hadoop/txtinput/7ldvc10.txt" - Aborting...
copyFromLocal: java.io.IOException: File /user/hadoop/txtinput/20417.txt
could only be replicated to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
java.io.IOException: File /user/hadoop/txtinput/7ldvc10.txt could only be
replicated to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
/usr/local/hadoop/bin$ ./hadoop jar ../hadoop-0.20.1-examples.jar
wordcount txtinput txtoutput
The last command just ends up sitting there doing nothing with no output.
Any help getting the nodes up and running would be appreciated.
Thanks,
Nick
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.
Posted by Sonal Goyal <so...@gmail.com>.
Yes, thanks Todd. I am looking to upgrade to 0.20.2.
Thanks and Regards,
Sonal
On Sat, Feb 13, 2010 at 11:07 PM, Todd Lipcon <to...@cloudera.com> wrote:
> Hi Sonal,
>
> Why are you using Hadoop 0.20.0? It's fairly old and there are lots of
> fixes in 0.20.1, and more in 0.20.2 which should be released any
> minute now.
>
> In particular, you're missing this change:
> https://issues.apache.org/jira/browse/HADOOP-5921
>
> which makes the JobTracker stubbornly wait for DFS to appear.
>
> I'd recommend using either (a) Apache 0.20.1, (b) Owen's rc of 0.20.2,
> or (c) Cloudera's 0.20.1 based build at
> http://archive.cloudera.com/cdh/2/hadoop-0.20.1+169.56.tar.gz which is
> 0.20.1 plus 225 extra patches (incl most of what's in 0.20.2).
>
> -Todd
>
> On Sat, Feb 13, 2010 at 8:35 AM, Sonal Goyal <so...@gmail.com>
> wrote:
> > Hi Aaron,
> >
> > I am on Hadoop 0.20.0 on Ubuntu, pseudo distributed mode. If I remove the
> > sleep time from my start-all.sh script, my jobtracker comes up
> momentarily
> > and then dies.
> >
> > Here is a capture of my commands:
> >
> > sgoyal@desktop:~/software/hadoop-0.20.0$ bin/hadoop namenode -format
> > 10/02/13 21:54:19 INFO namenode.NameNode: STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting NameNode
> > STARTUP_MSG: host = desktop/127.0.1.1
> > STARTUP_MSG: args = [-format]
> > STARTUP_MSG: version = 0.20.0
> > STARTUP_MSG: build =
> > https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r
> 763504;
> > compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
> > ************************************************************/
> > 10/02/13 21:54:19 DEBUG conf.Configuration: java.io.IOException: config()
> > at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:210)
> > at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:197)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:937)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:964)
> >
> > Re-format filesystem in /tmp/hadoop-sgoyal/dfs/name ? (Y or N) Y
> > 10/02/13 21:54:22 DEBUG security.UserGroupInformation: Unix Login:
> >
> sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
> > 10/02/13 21:54:22 INFO namenode.FSNamesystem:
> >
> fsOwner=sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
> > 10/02/13 21:54:22 INFO namenode.FSNamesystem: supergroup=supergroup
> > 10/02/13 21:54:22 INFO namenode.FSNamesystem: isPermissionEnabled=true
> > 10/02/13 21:54:22 INFO common.Storage: Image file of size 96 saved in 0
> > seconds.
> > 10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Preallocating Edit log,
> > current size 0
> > 10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Edit log size is now
> 1049088
> > written 512 bytes at offset 1048576
> > 10/02/13 21:54:22 INFO common.Storage: Storage directory
> > /tmp/hadoop-sgoyal/dfs/name has been successfully formatted.
> > 10/02/13 21:54:22 INFO namenode.NameNode: SHUTDOWN_MSG:
> > /************************************************************
> > SHUTDOWN_MSG: Shutting down NameNode at desktop/127.0.1.1
> > ************************************************************/
> >
> >
> > sgoyal@desktop:~/software/hadoop-0.20.0$ bin/start-all.sh
> > starting namenode, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-namenode-desktop.out
> > localhost: starting datanode, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-datanode-desktop.out
> > localhost: starting secondarynamenode, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-secondarynamenode-desktop.out
> > starting jobtracker, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-jobtracker-desktop.out
> > localhost: starting tasktracker, logging to
> >
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-tasktracker-desktop.out
> >
> > sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> > 26171 Jps
> > 26037 JobTracker
> > 25966 SecondaryNameNode
> > 25778 NameNode
> > 26130 TaskTracker
> > 25863 DataNode
> >
> > sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> > 26037 JobTracker
> > 25966 SecondaryNameNode
> > 26203 Jps
> > 25778 NameNode
> > 26130 TaskTracker
> > 25863 -- process information unavailable
> >
> > sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> > 26239 Jps
> > 26037 JobTracker
> > 25966 SecondaryNameNode
> > 25778 NameNode
> > 26130 TaskTracker
> >
> > sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> > 26037 JobTracker
> > 25966 SecondaryNameNode
> > 25778 NameNode
> > 26130 TaskTracker
> > 26252 Jps
> >
> > sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> > 26288 Jps
> > 25966 SecondaryNameNode
> > 25778 NameNode
> >
> > sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> > 25966 SecondaryNameNode
> > 25778 NameNode
> > 26298 Jps
> >
> > sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> > 26308 Jps
> > 25966 SecondaryNameNode
> > 25778 NameNode
> >
> > My jobtracker logs show:
> >
> > 2010-02-13 21:54:40,660 INFO org.apache.hadoop.mapred.JobTracker:
> > STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting JobTracker
> > STARTUP_MSG: host = desktop/127.0.1.1
> > STARTUP_MSG: args = []
> > STARTUP_MSG: version = 0.20.0
> > STARTUP_MSG: build =
> > https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r
> 763504;
> > compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
> > ************************************************************/
> > 2010-02-13 21:54:40,967 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> > Initializing RPC Metrics with hostName=JobTracker, port=9001
> > 2010-02-13 21:54:52,100 INFO org.mortbay.log: Logging to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 2010-02-13 21:54:52,358 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound
> > to port 50030
> > 2010-02-13 21:54:52,359 INFO org.mortbay.log: jetty-6.1.14
> > 2010-02-13 21:55:13,222 INFO org.mortbay.log: Started
> > SelectChannelConnector@0.0.0.0:50030
> > 2010-02-13 21:55:13,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > Initializing JVM Metrics with processName=JobTracker, sessionId=
> > 2010-02-13 21:55:13,229 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker
> > up at: 9001
> > 2010-02-13 21:55:13,229 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker
> > webserver: 50030
> > 2010-02-13 21:55:13,942 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2010-02-13 21:55:14,049 INFO org.apache.hadoop.hdfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /tmp/hadoop-sgoyal/mapred/system/jobtracker.info could only be
> replicated to
> > 0 nodes, instead of 1
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1256)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:739)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > at $Proxy4.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at $Proxy4.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2873)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2755)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2046)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2232)
> >
> > 2010-02-13 21:55:14,049 WARN org.apache.hadoop.hdfs.DFSClient:
> > NotReplicatedYetException sleeping /tmp/hadoop-sgoyal/mapred/system/
> > jobtracker.info retries left 4
> > 2010-02-13 21:55:14,459 INFO org.apache.hadoop.hdfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /tmp/hadoop-sgoyal/mapred/system/jobtracker.info could only be
> replicated to
> > 0 nodes, instead of 1
> >
> >
> > I suspected the dfs was not ready, and the sleep seems to solve this
> issue.
> > Look forward to hearing your take on this. Please feel free to let me
> know
> > if you need any other info.
> >
> > Thanks and Regards,
> > Sonal
> >
> >
> > On Sat, Feb 13, 2010 at 6:40 AM, Aaron Kimball <aa...@cloudera.com>
> wrote:
> >
> >> Sonal,
> >>
> >> Can I ask why you're sleeping between starting hdfs and mapreduce? I've
> >> never needed this in my own code. In general, Hadoop is pretty tolerant
> >> about starting daemons "out of order."
> >>
> >> If you need to wait for HDFS to be ready and come out of safe mode
> before
> >> launching a job, that's another story, but you can accomplish that with:
> >>
> >> $HADOOP_HOME/hadoop dfsadmin -safemode wait
> >>
> >> ... which will block until HDFS is ready for user commands in read/write
> >> mode.
> >> - Aaron
> >>
> >>
> >> On Fri, Feb 12, 2010 at 8:44 AM, Sonal Goyal <so...@gmail.com>
> >> wrote:
> >>
> >> > Hi
> >> >
> >> > I had faced a similar issue on Ubuntu and Hadoop 0.20 and modified the
> >> > start-all script to introduce a sleep time :
> >> >
> >> > bin=`dirname "$0"`
> >> > bin=`cd "$bin"; pwd`
> >> >
> >> > . "$bin"/hadoop-config.sh
> >> >
> >> > # start dfs daemons
> >> > "$bin"/start-dfs.sh --config $HADOOP_CONF_DIR
> >> > *echo 'sleeping'
> >> > sleep 60
> >> > echo 'awake'*
> >> > # start mapred daemons
> >> > "$bin"/start-mapred.sh --config $HADOOP_CONF_DIR
> >> >
> >> >
> >> > This seems to work. Please see if this works for you.
> >> > Thanks and Regards,
> >> > Sonal
> >> >
> >> >
> >> > On Thu, Feb 11, 2010 at 3:56 AM, E. Sammer <er...@lifeless.net> wrote:
> >> >
> >> > > On 2/10/10 5:19 PM, Nick Klosterman wrote:
> >> > >
> >> > >> @E.Sammer, no I don't *think* that it is part of another cluster.
> The
> >> > >> tutorial is for a single node cluster just as a initial set up to
> see
> >> if
> >> > >> you can get things up and running. I have reformatted the namenode
> >> > >> several times in my effort to get hadoop to work.
> >> > >>
> >> > >
> >> > > What I mean is that the data node, at some point, connected to your
> >> name
> >> > > node. If you reformat the name node, the data node must be wiped
> clean;
> >> > it's
> >> > > effectively trying to join a name node that no longer exists.
> >> > >
> >> > >
> >> > > --
> >> > > Eric Sammer
> >> > > eric@lifeless.net
> >> > > http://esammer.blogspot.com
> >> > >
> >> >
> >>
> >
>
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.
Posted by abhishek sharma <ab...@usc.edu>.
So Michael Noll's tutorial page has the following tips for the error
you are facing.
http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)#java.io.IOException:_Incompatible_namespaceIDs
Abhishek
On Wed, Feb 10, 2010 at 12:57 PM, Nick Klosterman
<nk...@ecn.purdue.edu> wrote:
> It appears I have incompatible namespaceIDs. Any thoughts on how to resolve
> that?
> This is what the full datanodes log is saying:
>
> 2010-02-10 15:25:09,125 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = potr134pc26/127.0.0.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.1
> STARTUP_MSG: build =
> http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r
> 810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC 2009
> ************************************************************/
> 2010-02-10 15:25:13,785 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in
> /home/hadoop/hadoop-datastore/hadoop-hadoop/dfs/data: namenode namespaceID =
> 2082816383; datanode namespaceID = 1109869136
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:233)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:148)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
>
> 2010-02-10 15:25:13,786 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at potr134pc26/127.0.0.1
> ************************************************************/
>
>
> On Wed, 10 Feb 2010, Nick Klosterman wrote:
>
>> I've been following Michael Noll's Single node cluster tutorial but am
>> unable to run the wordcount example successfully.
>>
>> It appears that I'm having some sort of problem involving the nodes. Using
>> copyFromLocal fails to replicate the data across 1 node.
>> When I try to look at the hadoop web interfaces I see that there aren't
>> any nodes (alive or dead) listed.
>>
>> After I start things this is what I get from dfsadmin -report
>> /usr/local/hadoop/bin$ ./hadoop dfsadmin -report
>> Configured Capacity: 0 (0 KB)
>> Present Capacity: 0 (0 KB)
>> DFS Remaining: 0 (0 KB)
>> DFS Used: 0 (0 KB)
>> DFS Used%: %
>> Under replicated blocks: 0
>> Blocks with corrupt replicas: 0
>> Missing blocks: 0
>>
>> -------------------------------------------------
>> Datanodes available: 0 (0 total, 0 dead)
>>
>>
>> Here are the commands I'm entering and the output of them:
>>
>> /usr/local/hadoop/bin$ ./start-all.sh
>> starting namenode, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
>> localhost: starting datanode, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
>> localhost: starting secondarynamenode, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
>> starting jobtracker, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
>> localhost: starting tasktracker, logging to
>> /usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
>>
>> /usr/local/hadoop/bin$ jps
>> 24440 SecondaryNameNode
>> 24626 TaskTracker
>> 24527 JobTracker
>> 24218 NameNode
>> 24725 Jps
>>
>> ---> I had all ready created the txtinput directory with ./hadoop dfs
>> -mkdir txtinput
>>
>> /usr/local/hadoop/bin$ ./hadoop dfs -copyFromLocal
>> /home/hadoop/Desktop/*.txt txtinput
>> 10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> /user/hadoop/txtinput/20417.txt could only be replicated to 0 nodes, instead
>> of 1
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:739)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>> at $Proxy0.addBlock(Unknown Source)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> at $Proxy0.addBlock(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
>>
>> 10/02/10 15:29:38 WARN hdfs.DFSClient: Error Recovery for block null bad
>> datanode[0] nodes == null
>> 10/02/10 15:29:38 WARN hdfs.DFSClient: Could not get block locations.
>> Source file "/user/hadoop/txtinput/20417.txt" - Aborting...
>> 10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> /user/hadoop/txtinput/7ldvc10.txt could only be replicated to 0 nodes,
>> instead of 1
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:739)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>> at $Proxy0.addBlock(Unknown Source)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> at $Proxy0.addBlock(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
>>
>> 10/02/10 15:29:38 WARN hdfs.DFSClient: Error Recovery for block null bad
>> datanode[0] nodes == null
>> 10/02/10 15:29:38 WARN hdfs.DFSClient: Could not get block locations.
>> Source file "/user/hadoop/txtinput/7ldvc10.txt" - Aborting...
>> copyFromLocal: java.io.IOException: File /user/hadoop/txtinput/20417.txt
>> could only be replicated to 0 nodes, instead of 1
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>
>> java.io.IOException: File /user/hadoop/txtinput/7ldvc10.txt could only be
>> replicated to 0 nodes, instead of 1
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>
>> /usr/local/hadoop/bin$ ./hadoop jar ../hadoop-0.20.1-examples.jar
>> wordcount txtinput txtoutput
>>
>> The last command just ends up sitting there doing nothing with no output.
>> Any help getting the nodes up and running would be appreciated.
>>
>> Thanks,
>> Nick
>>
>
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Sonal,
Why are you using Hadoop 0.20.0? It's fairly old and there are lots of
fixes in 0.20.1, and more in 0.20.2 which should be released any
minute now.
In particular, you're missing this change:
https://issues.apache.org/jira/browse/HADOOP-5921
which makes the JobTracker stubbornly wait for DFS to appear.
I'd recommend using either (a) Apache 0.20.1, (b) Owen's rc of 0.20.2,
or (c) Cloudera's 0.20.1 based build at
http://archive.cloudera.com/cdh/2/hadoop-0.20.1+169.56.tar.gz which is
0.20.1 plus 225 extra patches (incl most of what's in 0.20.2).
-Todd
On Sat, Feb 13, 2010 at 8:35 AM, Sonal Goyal <so...@gmail.com> wrote:
> Hi Aaron,
>
> I am on Hadoop 0.20.0 on Ubuntu, pseudo distributed mode. If I remove the
> sleep time from my start-all.sh script, my jobtracker comes up momentarily
> and then dies.
>
> Here is a capture of my commands:
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ bin/hadoop namenode -format
> 10/02/13 21:54:19 INFO namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = desktop/127.0.1.1
> STARTUP_MSG: args = [-format]
> STARTUP_MSG: version = 0.20.0
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r 763504;
> compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
> ************************************************************/
> 10/02/13 21:54:19 DEBUG conf.Configuration: java.io.IOException: config()
> at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:210)
> at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:197)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:937)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:964)
>
> Re-format filesystem in /tmp/hadoop-sgoyal/dfs/name ? (Y or N) Y
> 10/02/13 21:54:22 DEBUG security.UserGroupInformation: Unix Login:
> sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
> 10/02/13 21:54:22 INFO namenode.FSNamesystem:
> fsOwner=sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
> 10/02/13 21:54:22 INFO namenode.FSNamesystem: supergroup=supergroup
> 10/02/13 21:54:22 INFO namenode.FSNamesystem: isPermissionEnabled=true
> 10/02/13 21:54:22 INFO common.Storage: Image file of size 96 saved in 0
> seconds.
> 10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Preallocating Edit log,
> current size 0
> 10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Edit log size is now 1049088
> written 512 bytes at offset 1048576
> 10/02/13 21:54:22 INFO common.Storage: Storage directory
> /tmp/hadoop-sgoyal/dfs/name has been successfully formatted.
> 10/02/13 21:54:22 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at desktop/127.0.1.1
> ************************************************************/
>
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ bin/start-all.sh
> starting namenode, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-namenode-desktop.out
> localhost: starting datanode, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-datanode-desktop.out
> localhost: starting secondarynamenode, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-secondarynamenode-desktop.out
> starting jobtracker, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-jobtracker-desktop.out
> localhost: starting tasktracker, logging to
> /home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-tasktracker-desktop.out
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> 26171 Jps
> 26037 JobTracker
> 25966 SecondaryNameNode
> 25778 NameNode
> 26130 TaskTracker
> 25863 DataNode
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> 26037 JobTracker
> 25966 SecondaryNameNode
> 26203 Jps
> 25778 NameNode
> 26130 TaskTracker
> 25863 -- process information unavailable
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> 26239 Jps
> 26037 JobTracker
> 25966 SecondaryNameNode
> 25778 NameNode
> 26130 TaskTracker
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> 26037 JobTracker
> 25966 SecondaryNameNode
> 25778 NameNode
> 26130 TaskTracker
> 26252 Jps
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> 26288 Jps
> 25966 SecondaryNameNode
> 25778 NameNode
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> 25966 SecondaryNameNode
> 25778 NameNode
> 26298 Jps
>
> sgoyal@desktop:~/software/hadoop-0.20.0$ jps
> 26308 Jps
> 25966 SecondaryNameNode
> 25778 NameNode
>
> My jobtracker logs show:
>
> 2010-02-13 21:54:40,660 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host = desktop/127.0.1.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.0
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r 763504;
> compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
> ************************************************************/
> 2010-02-13 21:54:40,967 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=JobTracker, port=9001
> 2010-02-13 21:54:52,100 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2010-02-13 21:54:52,358 INFO org.apache.hadoop.http.HttpServer: Jetty bound
> to port 50030
> 2010-02-13 21:54:52,359 INFO org.mortbay.log: jetty-6.1.14
> 2010-02-13 21:55:13,222 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50030
> 2010-02-13 21:55:13,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=JobTracker, sessionId=
> 2010-02-13 21:55:13,229 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
> up at: 9001
> 2010-02-13 21:55:13,229 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
> webserver: 50030
> 2010-02-13 21:55:13,942 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
> up the system directory
> 2010-02-13 21:55:14,049 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /tmp/hadoop-sgoyal/mapred/system/jobtracker.info could only be replicated to
> 0 nodes, instead of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1256)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy4.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy4.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2873)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2755)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2046)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2232)
>
> 2010-02-13 21:55:14,049 WARN org.apache.hadoop.hdfs.DFSClient:
> NotReplicatedYetException sleeping /tmp/hadoop-sgoyal/mapred/system/
> jobtracker.info retries left 4
> 2010-02-13 21:55:14,459 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /tmp/hadoop-sgoyal/mapred/system/jobtracker.info could only be replicated to
> 0 nodes, instead of 1
>
>
> I suspected the dfs was not ready, and the sleep seems to solve this issue.
> Look forward to hearing your take on this. Please feel free to let me know
> if you need any other info.
>
> Thanks and Regards,
> Sonal
>
>
> On Sat, Feb 13, 2010 at 6:40 AM, Aaron Kimball <aa...@cloudera.com> wrote:
>
>> Sonal,
>>
>> Can I ask why you're sleeping between starting hdfs and mapreduce? I've
>> never needed this in my own code. In general, Hadoop is pretty tolerant
>> about starting daemons "out of order."
>>
>> If you need to wait for HDFS to be ready and come out of safe mode before
>> launching a job, that's another story, but you can accomplish that with:
>>
>> $HADOOP_HOME/hadoop dfsadmin -safemode wait
>>
>> ... which will block until HDFS is ready for user commands in read/write
>> mode.
>> - Aaron
>>
>>
>> On Fri, Feb 12, 2010 at 8:44 AM, Sonal Goyal <so...@gmail.com>
>> wrote:
>>
>> > Hi
>> >
>> > I had faced a similar issue on Ubuntu and Hadoop 0.20 and modified the
>> > start-all script to introduce a sleep time :
>> >
>> > bin=`dirname "$0"`
>> > bin=`cd "$bin"; pwd`
>> >
>> > . "$bin"/hadoop-config.sh
>> >
>> > # start dfs daemons
>> > "$bin"/start-dfs.sh --config $HADOOP_CONF_DIR
>> > *echo 'sleeping'
>> > sleep 60
>> > echo 'awake'*
>> > # start mapred daemons
>> > "$bin"/start-mapred.sh --config $HADOOP_CONF_DIR
>> >
>> >
>> > This seems to work. Please see if this works for you.
>> > Thanks and Regards,
>> > Sonal
>> >
>> >
>> > On Thu, Feb 11, 2010 at 3:56 AM, E. Sammer <er...@lifeless.net> wrote:
>> >
>> > > On 2/10/10 5:19 PM, Nick Klosterman wrote:
>> > >
>> > >> @E.Sammer, no I don't *think* that it is part of another cluster. The
>> > >> tutorial is for a single node cluster just as a initial set up to see
>> if
>> > >> you can get things up and running. I have reformatted the namenode
>> > >> several times in my effort to get hadoop to work.
>> > >>
>> > >
>> > > What I mean is that the data node, at some point, connected to your
>> name
>> > > node. If you reformat the name node, the data node must be wiped clean;
>> > it's
>> > > effectively trying to join a name node that no longer exists.
>> > >
>> > >
>> > > --
>> > > Eric Sammer
>> > > eric@lifeless.net
>> > > http://esammer.blogspot.com
>> > >
>> >
>>
>
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.
Posted by Sonal Goyal <so...@gmail.com>.
Hi Aaron,
I am on Hadoop 0.20.0 on Ubuntu, pseudo distributed mode. If I remove the
sleep time from my start-all.sh script, my jobtracker comes up momentarily
and then dies.
Here is a capture of my commands:
sgoyal@desktop:~/software/hadoop-0.20.0$ bin/hadoop namenode -format
10/02/13 21:54:19 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = desktop/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.0
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r 763504;
compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
************************************************************/
10/02/13 21:54:19 DEBUG conf.Configuration: java.io.IOException: config()
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:210)
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:197)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:937)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:964)
Re-format filesystem in /tmp/hadoop-sgoyal/dfs/name ? (Y or N) Y
10/02/13 21:54:22 DEBUG security.UserGroupInformation: Unix Login:
sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
10/02/13 21:54:22 INFO namenode.FSNamesystem:
fsOwner=sgoyal,sgoyal,adm,dialout,cdrom,audio,plugdev,fuse,lpadmin,admin,sambashare,mysql,cvsgroup
10/02/13 21:54:22 INFO namenode.FSNamesystem: supergroup=supergroup
10/02/13 21:54:22 INFO namenode.FSNamesystem: isPermissionEnabled=true
10/02/13 21:54:22 INFO common.Storage: Image file of size 96 saved in 0
seconds.
10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Preallocating Edit log,
current size 0
10/02/13 21:54:22 DEBUG namenode.FSNamesystem: Edit log size is now 1049088
written 512 bytes at offset 1048576
10/02/13 21:54:22 INFO common.Storage: Storage directory
/tmp/hadoop-sgoyal/dfs/name has been successfully formatted.
10/02/13 21:54:22 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at desktop/127.0.1.1
************************************************************/
sgoyal@desktop:~/software/hadoop-0.20.0$ bin/start-all.sh
starting namenode, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-namenode-desktop.out
localhost: starting datanode, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-datanode-desktop.out
localhost: starting secondarynamenode, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-secondarynamenode-desktop.out
starting jobtracker, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-jobtracker-desktop.out
localhost: starting tasktracker, logging to
/home/sgoyal/software/hadoop-0.20.0/bin/../logs/hadoop-sgoyal-tasktracker-desktop.out
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26171 Jps
26037 JobTracker
25966 SecondaryNameNode
25778 NameNode
26130 TaskTracker
25863 DataNode
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26037 JobTracker
25966 SecondaryNameNode
26203 Jps
25778 NameNode
26130 TaskTracker
25863 -- process information unavailable
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26239 Jps
26037 JobTracker
25966 SecondaryNameNode
25778 NameNode
26130 TaskTracker
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26037 JobTracker
25966 SecondaryNameNode
25778 NameNode
26130 TaskTracker
26252 Jps
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26288 Jps
25966 SecondaryNameNode
25778 NameNode
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
25966 SecondaryNameNode
25778 NameNode
26298 Jps
sgoyal@desktop:~/software/hadoop-0.20.0$ jps
26308 Jps
25966 SecondaryNameNode
25778 NameNode
My jobtracker logs show:
2010-02-13 21:54:40,660 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = desktop/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.0
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r 763504;
compiled by 'ndaley' on Thu Apr 9 05:18:40 UTC 2009
************************************************************/
2010-02-13 21:54:40,967 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=JobTracker, port=9001
2010-02-13 21:54:52,100 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2010-02-13 21:54:52,358 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50030
2010-02-13 21:54:52,359 INFO org.mortbay.log: jetty-6.1.14
2010-02-13 21:55:13,222 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50030
2010-02-13 21:55:13,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=JobTracker, sessionId=
2010-02-13 21:55:13,229 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
up at: 9001
2010-02-13 21:55:13,229 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
webserver: 50030
2010-02-13 21:55:13,942 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2010-02-13 21:55:14,049 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/tmp/hadoop-sgoyal/mapred/system/jobtracker.info could only be replicated to
0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1256)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy4.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2873)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2755)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2046)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2232)
2010-02-13 21:55:14,049 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping /tmp/hadoop-sgoyal/mapred/system/
jobtracker.info retries left 4
2010-02-13 21:55:14,459 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/tmp/hadoop-sgoyal/mapred/system/jobtracker.info could only be replicated to
0 nodes, instead of 1
I suspected the dfs was not ready, and the sleep seems to solve this issue.
Look forward to hearing your take on this. Please feel free to let me know
if you need any other info.
Thanks and Regards,
Sonal
On Sat, Feb 13, 2010 at 6:40 AM, Aaron Kimball <aa...@cloudera.com> wrote:
> Sonal,
>
> Can I ask why you're sleeping between starting hdfs and mapreduce? I've
> never needed this in my own code. In general, Hadoop is pretty tolerant
> about starting daemons "out of order."
>
> If you need to wait for HDFS to be ready and come out of safe mode before
> launching a job, that's another story, but you can accomplish that with:
>
> $HADOOP_HOME/hadoop dfsadmin -safemode wait
>
> ... which will block until HDFS is ready for user commands in read/write
> mode.
> - Aaron
>
>
> On Fri, Feb 12, 2010 at 8:44 AM, Sonal Goyal <so...@gmail.com>
> wrote:
>
> > Hi
> >
> > I had faced a similar issue on Ubuntu and Hadoop 0.20 and modified the
> > start-all script to introduce a sleep time :
> >
> > bin=`dirname "$0"`
> > bin=`cd "$bin"; pwd`
> >
> > . "$bin"/hadoop-config.sh
> >
> > # start dfs daemons
> > "$bin"/start-dfs.sh --config $HADOOP_CONF_DIR
> > *echo 'sleeping'
> > sleep 60
> > echo 'awake'*
> > # start mapred daemons
> > "$bin"/start-mapred.sh --config $HADOOP_CONF_DIR
> >
> >
> > This seems to work. Please see if this works for you.
> > Thanks and Regards,
> > Sonal
> >
> >
> > On Thu, Feb 11, 2010 at 3:56 AM, E. Sammer <er...@lifeless.net> wrote:
> >
> > > On 2/10/10 5:19 PM, Nick Klosterman wrote:
> > >
> > >> @E.Sammer, no I don't *think* that it is part of another cluster. The
> > >> tutorial is for a single node cluster just as a initial set up to see
> if
> > >> you can get things up and running. I have reformatted the namenode
> > >> several times in my effort to get hadoop to work.
> > >>
> > >
> > > What I mean is that the data node, at some point, connected to your
> name
> > > node. If you reformat the name node, the data node must be wiped clean;
> > it's
> > > effectively trying to join a name node that no longer exists.
> > >
> > >
> > > --
> > > Eric Sammer
> > > eric@lifeless.net
> > > http://esammer.blogspot.com
> > >
> >
>
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.
Posted by Aaron Kimball <aa...@cloudera.com>.
Sonal,
Can I ask why you're sleeping between starting hdfs and mapreduce? I've
never needed this in my own code. In general, Hadoop is pretty tolerant
about starting daemons "out of order."
If you need to wait for HDFS to be ready and come out of safe mode before
launching a job, that's another story, but you can accomplish that with:
$HADOOP_HOME/hadoop dfsadmin -safemode wait
... which will block until HDFS is ready for user commands in read/write
mode.
- Aaron
On Fri, Feb 12, 2010 at 8:44 AM, Sonal Goyal <so...@gmail.com> wrote:
> Hi
>
> I had faced a similar issue on Ubuntu and Hadoop 0.20 and modified the
> start-all script to introduce a sleep time :
>
> bin=`dirname "$0"`
> bin=`cd "$bin"; pwd`
>
> . "$bin"/hadoop-config.sh
>
> # start dfs daemons
> "$bin"/start-dfs.sh --config $HADOOP_CONF_DIR
> *echo 'sleeping'
> sleep 60
> echo 'awake'*
> # start mapred daemons
> "$bin"/start-mapred.sh --config $HADOOP_CONF_DIR
>
>
> This seems to work. Please see if this works for you.
> Thanks and Regards,
> Sonal
>
>
> On Thu, Feb 11, 2010 at 3:56 AM, E. Sammer <er...@lifeless.net> wrote:
>
> > On 2/10/10 5:19 PM, Nick Klosterman wrote:
> >
> >> @E.Sammer, no I don't *think* that it is part of another cluster. The
> >> tutorial is for a single node cluster just as a initial set up to see if
> >> you can get things up and running. I have reformatted the namenode
> >> several times in my effort to get hadoop to work.
> >>
> >
> > What I mean is that the data node, at some point, connected to your name
> > node. If you reformat the name node, the data node must be wiped clean;
> it's
> > effectively trying to join a name node that no longer exists.
> >
> >
> > --
> > Eric Sammer
> > eric@lifeless.net
> > http://esammer.blogspot.com
> >
>
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.
Posted by Sonal Goyal <so...@gmail.com>.
Hi
I had faced a similar issue on Ubuntu and Hadoop 0.20 and modified the
start-all script to introduce a sleep time :
bin=`dirname "$0"`
bin=`cd "$bin"; pwd`
. "$bin"/hadoop-config.sh
# start dfs daemons
"$bin"/start-dfs.sh --config $HADOOP_CONF_DIR
*echo 'sleeping'
sleep 60
echo 'awake'*
# start mapred daemons
"$bin"/start-mapred.sh --config $HADOOP_CONF_DIR
This seems to work. Please see if this works for you.
Thanks and Regards,
Sonal
On Thu, Feb 11, 2010 at 3:56 AM, E. Sammer <er...@lifeless.net> wrote:
> On 2/10/10 5:19 PM, Nick Klosterman wrote:
>
>> @E.Sammer, no I don't *think* that it is part of another cluster. The
>> tutorial is for a single node cluster just as a initial set up to see if
>> you can get things up and running. I have reformatted the namenode
>> several times in my effort to get hadoop to work.
>>
>
> What I mean is that the data node, at some point, connected to your name
> node. If you reformat the name node, the data node must be wiped clean; it's
> effectively trying to join a name node that no longer exists.
>
>
> --
> Eric Sammer
> eric@lifeless.net
> http://esammer.blogspot.com
>
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.
Posted by "E. Sammer" <er...@lifeless.net>.
On 2/10/10 5:19 PM, Nick Klosterman wrote:
> @E.Sammer, no I don't *think* that it is part of another cluster. The
> tutorial is for a single node cluster just as a initial set up to see if
> you can get things up and running. I have reformatted the namenode
> several times in my effort to get hadoop to work.
What I mean is that the data node, at some point, connected to your name
node. If you reformat the name node, the data node must be wiped clean;
it's effectively trying to join a name node that no longer exists.
--
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.
Posted by Nick Klosterman <nk...@ecn.purdue.edu>.
@E.Sammer, no I don't *think* that it is part of another cluster. The
tutorial is for a single node cluster just as a initial set up to see if
you can get things up and running. I have reformatted the namenode
several times in my effort to get hadoop to work.
@abishek
I tried the workaround you pointed me to no avail. I tried to modify
those directions since in the single node implementation I didn't have a
dfs.data.dir in hdfs-site.xml
My attempts at further debug
-----------------------------------------ATTEMPT AT FIXING THE DATANODES
PROBLEM
hadoop@potr134pc26:/usr/local/hadoop/bin$ rm -r
/usr/local/hadoop-datastore/
----NOW THERE IS NO HADOOP-DATASTORE FOLDER LOCALLY
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./hadoop namenode -format
10/02/10 16:33:50 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = potr134pc26/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.1
STARTUP_MSG: build =
http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r
810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC 2009
************************************************************/
Re-format filesystem in
/home/hadoop/hadoop-datastore/hadoop-hadoop/dfs/name ? (Y or N) Y
10/02/10 16:33:54 INFO namenode.FSNamesystem: fsOwner=hadoop,hadoop
10/02/10 16:33:54 INFO namenode.FSNamesystem: supergroup=supergroup
10/02/10 16:33:54 INFO namenode.FSNamesystem: isPermissionEnabled=true
10/02/10 16:33:54 INFO common.Storage: Image file of size 96 saved in 0
seconds.
10/02/10 16:33:54 INFO common.Storage: Storage directory
/home/hadoop/hadoop-datastore/hadoop-hadoop/dfs/name has been successfully
formatted.
10/02/10 16:33:54 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at potr134pc26/127.0.0.1
************************************************************/
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./start-all.sh
starting namenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
localhost: starting datanode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
localhost: starting secondarynamenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
starting jobtracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
localhost: starting tasktracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
hadoop@potr134pc26:/usr/local/hadoop/bin$ jps
27461 Jps
27354 TaskTracker
27158 SecondaryNameNode
27250 JobTracker
26923 NameNode
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./hadoop dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: %
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 0 (0 total, 0 dead)
---(((( AT THIS POINT WHEN I CHECKED THE LOG THE DATANODES STILL WASN'T UP
AND RUNNING)----------
mkdir /usr/local/hadoop-datastore
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./stop-all.sh
stopping jobtracker
localhost: stopping tasktracker
stopping namenode
localhost: no datanode to stop
localhost: stopping secondarynamenode
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./start-all.sh
starting namenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
localhost: starting datanode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
localhost: starting secondarynamenode, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
starting jobtracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
localhost: starting tasktracker, logging to
/usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
hadoop@potr134pc26:/usr/local/hadoop/bin$ jps
28038 NameNode
28536 Jps
28154 DataNode
28365 JobTracker
28470 TaskTracker
28272 SecondaryNameNode
./hadoop dfs -copyFromLocal /home/hadoop/Desktop/*.txt txtinput
copyFromLocal: `txtinput': specified destination directory doest not exist
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./hadoop dfs -mkdir txtinput
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./hadoop dfs -copyFromLocal
/home/hadoop/Desktop/*.txt txtinput
10/02/10 16:44:36 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/hadoop/txtinput/20417.txt could only be replicated to 0 nodes,
instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
10/02/10 16:44:36 WARN hdfs.DFSClient: Error Recovery for block null bad
datanode[0] nodes == null
10/02/10 16:44:36 WARN hdfs.DFSClient: Could not get block locations.
Source file "/user/hadoop/txtinput/20417.txt" - Aborting...
10/02/10 16:44:36 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/hadoop/txtinput/7ldvc10.txt could only be replicated to 0 nodes,
instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
10/02/10 16:44:36 WARN hdfs.DFSClient: Error Recovery for block null bad
datanode[0] nodes == null
10/02/10 16:44:36 WARN hdfs.DFSClient: Could not get block locations.
Source file "/user/hadoop/txtinput/7ldvc10.txt" - Aborting...
copyFromLocal: java.io.IOException: File /user/hadoop/txtinput/20417.txt
could only be replicated to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
java.io.IOException: File /user/hadoop/txtinput/7ldvc10.txt could only be
replicated to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
hadoop@potr134pc26:/usr/local/hadoop/bin$ ./hadoop dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: %
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 0 (0 total, 0 dead)
On Wed, 10 Feb 2010, E. Sammer wrote:
> On 2/10/10 3:57 PM, Nick Klosterman wrote:
>> It appears I have incompatible namespaceIDs. Any thoughts on how to
>> resolve that?
>> This is what the full datanodes log is saying:
>
> Was this data node part of a another DFS cluster at some point? It looks like
> you've reformatted the name node since the datanode connected to it. The
> datanode will refuse to connect to a namenode with a different namespaceId
> because the the data node would have blocks (possibly with the same ids) from
> another cluster. It's a stop gap safety mechanism. You'd have to destroy the
> data directory on the data node to "reinitialize" it so it picks up the new
> namespaceId from the name node at which point it will be allowed to connect.
>
> Just to be clear, this will also kill all data that was stored on the data
> node, so don't do this lightly.
>
> HTH.
> --
> Eric Sammer
> eric@lifeless.net
> http://esammer.blogspot.com
>
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.
Posted by "E. Sammer" <er...@lifeless.net>.
On 2/10/10 3:57 PM, Nick Klosterman wrote:
> It appears I have incompatible namespaceIDs. Any thoughts on how to
> resolve that?
> This is what the full datanodes log is saying:
Was this data node part of a another DFS cluster at some point? It looks
like you've reformatted the name node since the datanode connected to
it. The datanode will refuse to connect to a namenode with a different
namespaceId because the the data node would have blocks (possibly with
the same ids) from another cluster. It's a stop gap safety mechanism.
You'd have to destroy the data directory on the data node to
"reinitialize" it so it picks up the new namespaceId from the name node
at which point it will be allowed to connect.
Just to be clear, this will also kill all data that was stored on the
data node, so don't do this lightly.
HTH.
--
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.
Posted by Nick Klosterman <nk...@ecn.purdue.edu>.
It appears I have incompatible namespaceIDs. Any thoughts on how to
resolve that?
This is what the full datanodes log is saying:
2010-02-10 15:25:09,125 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = potr134pc26/127.0.0.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.1
STARTUP_MSG: build =
http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r
810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC 2009
************************************************************/
2010-02-10 15:25:13,785 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
Incompatible namespaceIDs in
/home/hadoop/hadoop-datastore/hadoop-hadoop/dfs/data: namenode namespaceID
= 2082816383; datanode namespaceID = 1109869136
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:233)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:148)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
2010-02-10 15:25:13,786 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at potr134pc26/127.0.0.1
************************************************************/
On Wed, 10 Feb 2010, Nick Klosterman wrote:
> I've been following Michael Noll's Single node cluster tutorial but am
> unable to run the wordcount example successfully.
>
> It appears that I'm having some sort of problem involving the nodes. Using
> copyFromLocal fails to replicate the data across 1 node.
> When I try to look at the hadoop web interfaces I see that there aren't any
> nodes (alive or dead) listed.
>
> After I start things this is what I get from dfsadmin -report
> /usr/local/hadoop/bin$ ./hadoop dfsadmin -report
> Configured Capacity: 0 (0 KB)
> Present Capacity: 0 (0 KB)
> DFS Remaining: 0 (0 KB)
> DFS Used: 0 (0 KB)
> DFS Used%: %
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 0 (0 total, 0 dead)
>
>
> Here are the commands I'm entering and the output of them:
>
> /usr/local/hadoop/bin$ ./start-all.sh
> starting namenode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
> localhost: starting datanode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
> localhost: starting secondarynamenode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
> starting jobtracker, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
> localhost: starting tasktracker, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
>
> /usr/local/hadoop/bin$ jps
> 24440 SecondaryNameNode
> 24626 TaskTracker
> 24527 JobTracker
> 24218 NameNode
> 24725 Jps
>
> ---> I had all ready created the txtinput directory with ./hadoop dfs -mkdir
> txtinput
>
> /usr/local/hadoop/bin$ ./hadoop dfs -copyFromLocal /home/hadoop/Desktop/*.txt
> txtinput
> 10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/hadoop/txtinput/20417.txt could only be replicated to 0 nodes, instead
> of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
>
> 10/02/10 15:29:38 WARN hdfs.DFSClient: Error Recovery for block null bad
> datanode[0] nodes == null
> 10/02/10 15:29:38 WARN hdfs.DFSClient: Could not get block locations. Source
> file "/user/hadoop/txtinput/20417.txt" - Aborting...
> 10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/hadoop/txtinput/7ldvc10.txt could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
>
> 10/02/10 15:29:38 WARN hdfs.DFSClient: Error Recovery for block null bad
> datanode[0] nodes == null
> 10/02/10 15:29:38 WARN hdfs.DFSClient: Could not get block locations. Source
> file "/user/hadoop/txtinput/7ldvc10.txt" - Aborting...
> copyFromLocal: java.io.IOException: File /user/hadoop/txtinput/20417.txt
> could only be replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> java.io.IOException: File /user/hadoop/txtinput/7ldvc10.txt could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> /usr/local/hadoop/bin$ ./hadoop jar ../hadoop-0.20.1-examples.jar wordcount
> txtinput txtoutput
>
> The last command just ends up sitting there doing nothing with no output.
> Any help getting the nodes up and running would be appreciated.
>
> Thanks,
> Nick
>
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.
Posted by "E. Sammer" <er...@lifeless.net>.
Nick:
It appears that the datanode daemon isn't running.
> /usr/local/hadoop/bin$ jps
> 24440 SecondaryNameNode
> 24626 TaskTracker
> 24527 JobTracker
> 24218 NameNode
> 24725 Jps
There's no process for DataNode. This is the process that is responsible
for storing blocks. In other words, no data node, no storage. The
namenode only manages the file system image and metadata.
Check the logs $HADOOP_HOME/logs/*datanode*.log for errors in start up.
Hope this helps.
On 2/10/10 3:42 PM, Nick Klosterman wrote:
> I've been following Michael Noll's Single node cluster tutorial but am
> unable to run the wordcount example successfully.
>
> It appears that I'm having some sort of problem involving the nodes.
> Using copyFromLocal fails to replicate the data across 1 node.
> When I try to look at the hadoop web interfaces I see that there aren't
> any nodes (alive or dead) listed.
>
> After I start things this is what I get from dfsadmin -report
> /usr/local/hadoop/bin$ ./hadoop dfsadmin -report
> Configured Capacity: 0 (0 KB)
> Present Capacity: 0 (0 KB)
> DFS Remaining: 0 (0 KB)
> DFS Used: 0 (0 KB)
> DFS Used%: %
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 0 (0 total, 0 dead)
>
>
> Here are the commands I'm entering and the output of them:
>
> /usr/local/hadoop/bin$ ./start-all.sh
> starting namenode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-potr134pc26.out
> localhost: starting datanode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-potr134pc26.out
> localhost: starting secondarynamenode, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-potr134pc26.out
>
> starting jobtracker, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-potr134pc26.out
> localhost: starting tasktracker, logging to
> /usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-potr134pc26.out
>
> /usr/local/hadoop/bin$ jps
> 24440 SecondaryNameNode
> 24626 TaskTracker
> 24527 JobTracker
> 24218 NameNode
> 24725 Jps
>
> ---> I had all ready created the txtinput directory with ./hadoop dfs
> -mkdir txtinput
>
> /usr/local/hadoop/bin$ ./hadoop dfs -copyFromLocal
> /home/hadoop/Desktop/*.txt txtinput
> 10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/hadoop/txtinput/20417.txt could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>
> at $Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
>
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
>
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
>
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
>
>
> 10/02/10 15:29:38 WARN hdfs.DFSClient: Error Recovery for block null bad
> datanode[0] nodes == null
> 10/02/10 15:29:38 WARN hdfs.DFSClient: Could not get block locations.
> Source file "/user/hadoop/txtinput/20417.txt" - Aborting...
> 10/02/10 15:29:38 WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/hadoop/txtinput/7ldvc10.txt could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>
> at $Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
>
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
>
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
>
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
>
>
> 10/02/10 15:29:38 WARN hdfs.DFSClient: Error Recovery for block null bad
> datanode[0] nodes == null
> 10/02/10 15:29:38 WARN hdfs.DFSClient: Could not get block locations.
> Source file "/user/hadoop/txtinput/7ldvc10.txt" - Aborting...
> copyFromLocal: java.io.IOException: File /user/hadoop/txtinput/20417.txt
> could only be replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> java.io.IOException: File /user/hadoop/txtinput/7ldvc10.txt could only
> be replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> /usr/local/hadoop/bin$ ./hadoop jar ../hadoop-0.20.1-examples.jar
> wordcount txtinput txtoutput
>
> The last command just ends up sitting there doing nothing with no output.
> Any help getting the nodes up and running would be appreciated.
>
> Thanks,
> Nick
>
--
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com
Re: Ubuntu Single Node Tutorial failure. No live or dead nodes.
Posted by Allen Wittenauer <aw...@linkedin.com>.
On 2/10/10 12:42 PM, "Nick Klosterman" <nk...@ecn.purdue.edu> wrote:
> I've been following Michael Noll's Single node cluster tutorial but am
> unable to run the wordcount example successfully.
>
> It appears that I'm having some sort of problem involving the nodes. Using
> copyFromLocal fails to replicate the data across 1 node.
> When I try to look at the hadoop web interfaces I see that there aren't
> any nodes (alive or dead) listed.
>
> After I start things this is what I get from dfsadmin -report
> /usr/local/hadoop/bin$ ./hadoop dfsadmin -report
> Configured Capacity: 0 (0 KB)
> Present Capacity: 0 (0 KB)
> DFS Remaining: 0 (0 KB)
> DFS Used: 0 (0 KB)
> DFS Used%: %
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
You have no datanodes. Check the *out and *log files for the datanode
processes.