You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Kartashov, Andy" <An...@mpac.ca> on 2012/10/26 16:40:43 UTC
RE: datanode daemon SOLVED`
Gents,
Need to share with you my embarrassment... Solved this issue.. How?
Well, while following the installation instructions I thought I installed all the daemons, but, after checking the init.d folder I could not find hadoop-hdfs-datanode script so (thinking I acciddentslly deleted it) I merely scp'ied the script from another node.
I've tried in vain to start that node for at least 13 hours until, I went on installing hadoop on a new node, I realised that I missed data-node installation all together.
I was supposed to run:
sudo yum install hadoop-0.20-mapreduce-tasktracker hadoop-hdfs-datanode
but I ran only
sudo yum install hadoop-0.20-mapreduce-tasktracker
After installing datanode and reformatting the namespace, datanode started like a new engine.
Silly me. Oh well. :) Calm seas do not make good sailors.
AK47
From: Kartashov, Andy
Sent: Thursday, October 25, 2012 3:40 PM
To: user@hadoop.apache.org
Subject: datanode daemon
Guys,
I finally solved ALL the Errors: in ...datanode*.log after trying to start the node with "service datanode start".
The errors were:
- conflicting NN DD ids - solved through reformatting NN.
- could not connect to 127.0.0.1:8020 - Connection refused - solved through correcting a typo inside hdfs-site.xml under dfs.namenode.http-address; somehow had the default value i/o localhost. (Running pseudo-mode)
- conf was pointing to the wrong sLink - solved by running alternatives -set hadoop-conf <conf.myconf>
However, when I run "service -status-all", still see that datanode [FAILED] message. All others, NN, SNN, JT, TT are running [OK].
1. Starting daemons, all seems OK:
Starting Hadoop datanode: [ OK ]
starting datanode, logging to /home/hadoop/logs/hadoop-root-datanode-ip-10-204-47-138.out
Starting Hadoop namenode: [ OK ]
starting namenode, logging to /home/hadoop/logs/hadoop-hdfs-namenode-ip-10-204-47-138.out
Starting Hadoop secondarynamenode: [ OK ]
starting secondarynamenode, logging to /home/hadoop/logs/hadoop-hdfs-secondarynamenode-ip-10-204-47-138.out
2.
running service -status-all command and get:
Hadoop datanode is not running [FAILED]
Hadoop namenode is running [ OK ]
Hadoop secondarynamenode is running [ OK ]
3.
Here is log file on DN:
2012-10-25 15:33:37,554 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = ip-10-204-47-138.ec2.internal/10.204.47.138
STARTUP_MSG: args = []
STARTUP_MSG: version = 2.0.0-cdh4.1.1
STARTUP_MSG: classpath = /etc/ha..........
...............................
..............................
2012-10-25 15:33:38,098 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hadoop/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration.
2012-10-25 15:33:41,589 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2012-10-25 15:33:42,125 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2012-10-25 15:33:42,125 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
2012-10-25 15:33:42,204 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Configured hostname is ip-10-204-47-138.ec2.internal
2012-10-25 15:33:42,319 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened streaming server at /0.0.0.0:50010
2012-10-25 15:33:42,323 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
2012-10-25 15:33:42,412 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-10-25 15:33:42,603 INFO org.apache.hadoop.http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-10-25 15:33:42,607 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
2012-10-25 15:33:42,607 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2012-10-25 15:33:42,607 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2012-10-25 15:33:42,682 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 0.0.0.0:50075
2012-10-25 15:33:42,690 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dfs.webhdfs.enabled = false
2012-10-25 15:33:42,690 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
2012-10-25 15:33:42,690 INFO org.mortbay.log: jetty-6.1.26.cloudera.2
2012-10-25 15:33:43,601 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075<mailto:SelectChannelConnector@0.0.0.0:50075>
2012-10-25 15:33:43,787 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
2012-10-25 15:33:43,905 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:50020
2012-10-25 15:33:43,917 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null
2012-10-25 15:33:43,943 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default>
2012-10-25 15:33:43,950 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hadoop/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration.
2012-10-25 15:33:43,958 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (storage id unknown) service to localhost/127.0.0.1:8020 starting to offer service
2012-10-25 15:33:44,297 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2012-10-25 15:33:44,304 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2012-10-25 15:33:45,551 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2012-10-25 15:33:46,605 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2012-10-25 15:33:47,865 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2012-10-25 15:33:48,945 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2012-10-25 15:33:49,946 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2012-10-25 15:33:50,947 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2012-10-25 15:33:51,976 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2012-10-25 15:33:52,997 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2012-10-25 15:33:54,047 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2012-10-25 15:33:57,579 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/hadoop/dfs/data/in_use.lock acquired by nodename 21886@ip-10-204-47-138.ec2.internal<ma...@ip-10-204-47-138.ec2.internal>
2012-10-25 15:33:59,279 INFO org.apache.hadoop.hdfs.server.common.Storage: Locking is disabled
2012-10-25 15:33:59,297 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Setting up storage: nsid=587246899;bpid=BP-443540453-10.204.47.138-1351175525331;lv=-40;nsInfo=lv=-40;cid=CID-55808532-08a8-447c-b5e7-e609f6c546ee;nsid=587246899;c=0;bpid=BP-443540453-10.204.47.138-1351175525331
2012-10-25 15:33:59,457 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added volume - /home/hadoop/dfs/data/current
2012-10-25 15:33:59,583 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Registered FSDatasetState MBean
2012-10-25 15:33:59,680 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting at 1351195722679 with interval 21600000
2012-10-25 15:33:59,688 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-443540453-10.204.47.138-1351175525331
2012-10-25 15:34:00,339 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-443540453-10.204.47.138-1351175525331 (storage id DS-1489896713-10.204.47.138-50010-1351175686521) service to localhost/127.0.0.1:8020 beginning handshake with NN
2012-10-25 15:34:00,485 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-443540453-10.204.47.138-1351175525331 (storage id DS-1489896713-10.204.47.138-50010-1351175686521) service to localhost/127.0.0.1:8020 successfully registered with NN
2012-10-25 15:34:00,491 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode localhost/127.0.0.1:8020 using DELETEREPORT_INTERVAL of 300000 msec BLOCKREPORT_INTERVAL of 21600000msec Initial delay: 0msec; heartBeatInterval=3000
2012-10-25 15:34:01,106 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Namenode Block pool BP-443540453-10.204.47.138-1351175525331 (storage id DS-1489896713-10.204.47.138-50010-1351175686521) service to localhost/127.0.0.1:8020 trying to claim ACTIVE state with txid=2697
2012-10-25 15:34:01,106 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode Block pool BP-443540453-10.204.47.138-1351175525331 (storage id DS-1489896713-10.204.47.138-50010-1351175686521) service to localhost/127.0.0.1:8020
2012-10-25 15:34:01,483 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 30 blocks took 8 msec to generate and 369 msecs for RPC and NN processing
2012-10-25 15:34:01,483 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: sent block report, processed command:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@d2f41a5
2012-10-25 15:34:01,496 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Periodic Block Verification Scanner initialized with interval 504 hours for block pool BP-443540453-10.204.47.138-1351175525331.
2012-10-25 15:34:01,516 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Added bpid=BP-443540453-10.204.47.138-1351175525331 to blockPoolScannerMap, new size=1
2012-10-25 15:34:04,939 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-2780693186250942847_1214
2012-10-25 15:34:04,962 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_3001053775353836209_1049
2012-10-25 15:34:05,081 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-5830536650401518708_1776
2012-10-25 15:34:05,100 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-7796621539450888414_1161
2012-10-25 15:34:05,110 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_3113923085061591346_1318
2012-10-25 15:34:05,111 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-3360339171714568163_1477
2012-10-25 15:34:05,111 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-8153671645708824194_1779
2012-10-25 15:34:05,112 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-1321321546165212517_1002
2012-10-25 15:34:05,116 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-7769865739291371890_1479
Rgds,
AK47
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel