You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by "Kilbride, James P." <Ja...@gd-ais.com> on 2010/06/15 19:09:46 UTC
IPC connections failing
I've tried to move my system from pseudo distributed to full
distributed(with it being the first node in the full cluster I'll be
setting up) but the components don't seem to be talking to each other
and I find logs filled with ipc.Client connection failures.
Here's the basic log info:
DataNode:
2010-06-15 12:42:42,824 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:49,335 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:42:50,343 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:42:51,354 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:42:53,641 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:42:54,733 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:42:55,737 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:42:56,824 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:42:57,827 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:42:58,831 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:42:59,833 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:42:59,836 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...
2010-06-15 12:43:01,841 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:43:02,844 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:43:03,846 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:43:04,849 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:43:05,852 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:06,855 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:07,858 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:08,883 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:09,970 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...
JobTracker:
2010-06-15 12:42:52,836 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:53,350 INFO org.apache.hadoop.mapred.JobTracker:
Scheduler configured with (memSizeForMapSlotOnJT,
memSizeForReduceSlotOnJT, limitMaxMemForMapTasks,
limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2010-06-15 12:42:53,815 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=JobTracker, port=54311
2010-06-15 12:42:53,986 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2010-06-15 12:42:55,381 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is
-1. Opening the listener on 50030
2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50030
webServer.getConnectors()[0].getLocalPort() returned 50030
2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer: Jetty
bound to port 50030
2010-06-15 12:42:55,402 INFO org.mortbay.log: jetty-6.1.14
2010-06-15 12:42:56,802 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50030
2010-06-15 12:42:56,805 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=JobTracker, sessionId=
2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker up at: 54311
2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker webserver: 50030
2010-06-15 12:42:58,076 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:42:59,080 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:43:00,083 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:43:01,086 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:43:02,089 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:03,091 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:04,095 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:05,174 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:06,177 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:07,180 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:07,185 INFO org.apache.hadoop.mapred.JobTracker:
problem cleaning system directory: null
java.net.ConnectException: Call to /10.28.208.118:54310 failed on
connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:767)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileS
ystem.java:82)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at
org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1665)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
at
org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.ja
va:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
at
org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:860)
at org.apache.hadoop.ipc.Client.call(Client.java:720)
... 16 more
NameNode:
2010-06-15 12:42:38,714 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:38,922 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=NameNode, port=54310
2010-06-15 12:42:38,932 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at:
centoshadoop/10.28.208.118:54310
2010-06-15 12:42:38,938 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=NameNode, sessionId=null
2010-06-15 12:42:38,941 INFO
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics:
Initializing NameNodeMeterics using context
object:org.apache.hadoop.metrics.spi.NullContext
2010-06-15 12:42:39,140 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
fsOwner=hadoop,users,hadoop
2010-06-15 12:42:39,141 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
supergroup=supergroup
2010-06-15 12:42:39,141 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
isPermissionEnabled=true
2010-06-15 12:42:39,156 INFO
org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
Initializing FSNamesystemMetrics using context
object:org.apache.hadoop.metrics.spi.NullContext
2010-06-15 12:42:39,161 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean
2010-06-15 12:42:39,381 INFO
org.apache.hadoop.hdfs.server.common.Storage: Number of files = 28
2010-06-15 12:42:39,404 INFO
org.apache.hadoop.hdfs.server.common.Storage: Number of files under
construction = 0
2010-06-15 12:42:39,404 INFO
org.apache.hadoop.hdfs.server.common.Storage: Image file of size 2780
loaded in 0 seconds.
2010-06-15 12:42:39,429 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:
java.lang.NumberFormatException: For input string: ""
at
java.lang.NumberFormatException.forInputString(NumberFormatException.jav
a:48)
at java.lang.Long.parseLong(Long.java:431)
at java.lang.Long.parseLong(Long.java:468)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.readLong(FSEditLog.java
:1273)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
ava:670)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:
992)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:
812)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSI
mage.java:364)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirecto
ry.java:87)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesys
tem.java:311)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
java:292)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java
:201)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279
)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
java:956)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
SecondaryNameNode:
2010-06-15 12:42:53,227 INFO
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:53,442 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=SecondaryNameNode,
sessionId=null
2010-06-15 12:42:55,651 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:42:56,659 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:42:57,662 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:42:58,665 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:42:59,668 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:00,671 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:01,674 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:02,677 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:03,679 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:04,682 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:04,685 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...
2010-06-15 12:43:06,691 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:43:07,694 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:43:08,697 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:43:09,700 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:43:10,703 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:11,706 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:12,709 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:13,712 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:14,714 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:15,717 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:15,718 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...
TaskTracker:
2010-06-15 12:42:57,270 INFO org.apache.hadoop.mapred.TaskTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:57,854 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2010-06-15 12:42:58,328 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is
-1. Opening the listener on 50060
2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50060
webServer.getConnectors()[0].getLocalPort() returned 50060
2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer: Jetty
bound to port 50060
2010-06-15 12:42:58,345 INFO org.mortbay.log: jetty-6.1.14
2010-06-15 12:42:58,852 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50060
2010-06-15 12:42:58,860 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=TaskTracker, sessionId=
2010-06-15 12:42:58,882 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=TaskTracker, port=50758
2010-06-15 12:42:58,960 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2010-06-15 12:42:58,962 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 50758: starting
2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 50758: starting
2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 50758: starting
2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
TaskTracker up at: localhost.localdomain/127.0.0.1:50758
2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
Starting tracker
tracker_centoshadoop:localhost.localdomain/127.0.0.1:50758
2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 50758: starting
2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 50758: starting
The config files are:
core-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop-0.20.2/tmp/dir/hadoop-${user.name}</value>
<description> A base for the other temporary
directories.</description>
<final>true</final>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://10.28.208.118:54310</value>
<description> The name of the default file system. A URI whose scheme
and auth
ority determine the filesystem implementation. The uri's scheme
determines the c
onfig property (fs.SCHEME.impl) name the FileSystem implementation
class. The ur
i's authority is used to determine the host, port, etc for a
filesystem</description>
<final>true</final>
</property>
</configuration>
hadoop-env.sh
# Set Hadoop-specific environment variables here.
# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.
# The java implementation to use. Required.
export JAVA_HOME=/usr/java/default
# The hadoop home directory
export HADOOP_HOME=/hadoop-0.20.2
# Extra Java CLASSPATH elements. Optional.
export
HADOOP_CLASSPATH=/hbase-0.20.4/lib/zookeeper-3.2.2.jar:/habase-0.20.4/hb
ase-0.20.4-test.jar:/hbase-0.20.4/hbase-0.20.4.jar:/hbase-0.20.4/conf
# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000
# Extra Java runtime options. Empty by default.
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS
# Extra ssh options. Empty by default.
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o
SendEnv=HADOOP_CONF_DIR"
# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
# File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
# host:path where hadoop code should be rsync'd from. Unset by default.
# export HADOOP_MASTER=master:/home/$USER/src/hadoop
# Seconds to sleep between slave commands. Unset by default. This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=0.1
# The directory where pid files are stored. /tmp by default.
# export HADOOP_PID_DIR=/var/hadoop/pids
# A string representing this instance of hadoop. $USER by default.
# export HADOOP_IDENT_STRING=$USER
# The scheduling priority for daemon processes. See 'man nice'.
# export HADOOP_NICENESS=10
hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>22</value>
<description> Default block replication. The actual number of
replications can be specified when the file is created. Default is used
if not specified at creation time.</description>
<final>true</final>
</property>
<property>
<name>dfs.data.dir</name>
<value>/hadoop-0.20.2/hadoopDataStore/data</value>
<description>DFS Data Directory</description>
<final>true</final>
</property>
</configuration>
masters:
10.28.208.118
slaves:
10.28.208.118
(And because I know it'll be asked)
hosts:
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
10.28.208.118 centoshadoop centoshadoop.soa.gd-ais.com
Why is this failing to stand up? I'd like to get this one node working
before I stand up any other data nodes.(Firewall, btw, is turned off)
James Kilbride
RE: IPC connections failing
Posted by "Kilbride, James P." <Ja...@gd-ais.com>.
How do I repair that and prevent it from happening again?
James Kilbride
-----Original Message-----
From: Dmytro Molkov [mailto:dms@facebook.com]
Sent: Tuesday, June 15, 2010 3:00 PM
To: general@hadoop.apache.org
Subject: Re: IPC connections failing
Based on the log of the namenode I do not think it started successfully.
Seems like your edits log is corrupted
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
ava:670)
So all other parts of the system could not connect to the namenode
because it was down, not because of IPC problems.
On Jun 15, 2010, at 10:11 AM, Kilbride, James P. wrote:
> Forgot to add on mapred-site.xml:
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
> <name>mapred.job.tracker</name>
> <value>hdfs://10.28.208.118:54311</value>
> <description> The host and port that the MapReduce job tracker runs
> at. If "local", then the jobs are run in-process as a single map and
> reduce task.</description>
> <final>true</final>
> </property>
>
> </configuration>
>
> -----Original Message-----
> From: Kilbride, James P. [mailto:James.Kilbride@gd-ais.com]
> Sent: Tuesday, June 15, 2010 1:10 PM
> To: general@hadoop.apache.org
> Subject: IPC connections failing
>
> I've tried to move my system from pseudo distributed to full
> distributed(with it being the first node in the full cluster I'll be
> setting up) but the components don't seem to be talking to each other
> and I find logs filled with ipc.Client connection failures.
>
> Here's the basic log info:
> DataNode:
> 2010-06-15 12:42:42,824 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:49,335 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:42:50,343 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:42:51,354 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:42:53,641 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:42:54,733 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:42:55,737 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:42:56,824 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:42:57,827 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:42:58,831 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:42:59,833 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:42:59,836 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
> 2010-06-15 12:43:01,841 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:43:02,844 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:43:03,846 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:43:04,849 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:43:05,852 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:06,855 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:07,858 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:08,883 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:09,970 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
>
> JobTracker:
> 2010-06-15 12:42:52,836 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:53,350 INFO org.apache.hadoop.mapred.JobTracker:
> Scheduler configured with (memSizeForMapSlotOnJT,
> memSizeForReduceSlotOnJT, limitMaxMemForMapTasks,
> limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2010-06-15 12:42:53,815 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=JobTracker, port=54311
> 2010-06-15 12:42:53,986 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2010-06-15 12:42:55,381 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open()
is
> -1. Opening the listener on 50030
> 2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50030
> webServer.getConnectors()[0].getLocalPort() returned 50030
> 2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50030
> 2010-06-15 12:42:55,402 INFO org.mortbay.log: jetty-6.1.14
> 2010-06-15 12:42:56,802 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50030
> 2010-06-15 12:42:56,805 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=JobTracker, sessionId=
> 2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker up at: 54311
> 2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker webserver: 50030
> 2010-06-15 12:42:58,076 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:42:59,080 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:43:00,083 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:43:01,086 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:43:02,089 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:03,091 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:04,095 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:05,174 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:06,177 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:07,180 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:07,185 INFO org.apache.hadoop.mapred.JobTracker:
> problem cleaning system directory: null
> java.net.ConnectException: Call to /10.28.208.118:54310 failed on
> connection exception: java.net.ConnectException: Connection refused
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:767)
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy4.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> at
> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
> at
>
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileS
> ystem.java:82)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
> at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
> at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
> at
> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1665)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> at
> org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
>
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.ja
> va:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at
>
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
> at
> org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:860)
> at org.apache.hadoop.ipc.Client.call(Client.java:720)
> ... 16 more
>
> NameNode:
> 2010-06-15 12:42:38,714 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:38,922 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=NameNode, port=54310
> 2010-06-15 12:42:38,932 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at:
> centoshadoop/10.28.208.118:54310
> 2010-06-15 12:42:38,938 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=NameNode, sessionId=null
> 2010-06-15 12:42:38,941 INFO
> org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics:
> Initializing NameNodeMeterics using context
> object:org.apache.hadoop.metrics.spi.NullContext
> 2010-06-15 12:42:39,140 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> fsOwner=hadoop,users,hadoop
> 2010-06-15 12:42:39,141 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> supergroup=supergroup
> 2010-06-15 12:42:39,141 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled=true
> 2010-06-15 12:42:39,156 INFO
> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
> Initializing FSNamesystemMetrics using context
> object:org.apache.hadoop.metrics.spi.NullContext
> 2010-06-15 12:42:39,161 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStatusMBean
> 2010-06-15 12:42:39,381 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Number of files = 28
> 2010-06-15 12:42:39,404 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Number of files under
> construction = 0
> 2010-06-15 12:42:39,404 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Image file of size 2780
> loaded in 0 seconds.
> 2010-06-15 12:42:39,429 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode:
> java.lang.NumberFormatException: For input string: ""
> at
>
java.lang.NumberFormatException.forInputString(NumberFormatException.jav
> a:48)
> at java.lang.Long.parseLong(Long.java:431)
> at java.lang.Long.parseLong(Long.java:468)
> at
>
org.apache.hadoop.hdfs.server.namenode.FSEditLog.readLong(FSEditLog.java
> :1273)
> at
>
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
> ava:670)
> at
>
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:
> 992)
> at
>
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:
> 812)
> at
>
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSI
> mage.java:364)
> at
>
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirecto
> ry.java:87)
> at
>
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesys
> tem.java:311)
> at
>
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
> java:292)
> at
>
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java
> :201)
> at
>
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279
> )
> at
>
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
> java:956)
> at
>
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>
> SecondaryNameNode:
> 2010-06-15 12:42:53,227 INFO
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting SecondaryNameNode
> STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:53,442 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=SecondaryNameNode,
> sessionId=null
> 2010-06-15 12:42:55,651 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:42:56,659 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:42:57,662 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:42:58,665 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:42:59,668 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:00,671 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:01,674 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:02,677 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:03,679 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:04,682 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:04,685 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
> 2010-06-15 12:43:06,691 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:43:07,694 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:43:08,697 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:43:09,700 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:43:10,703 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:11,706 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:12,709 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:13,712 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:14,714 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:15,717 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:15,718 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
>
> TaskTracker:
> 2010-06-15 12:42:57,270 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:57,854 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2010-06-15 12:42:58,328 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open()
is
> -1. Opening the listener on 50060
> 2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50060
> 2010-06-15 12:42:58,345 INFO org.mortbay.log: jetty-6.1.14
> 2010-06-15 12:42:58,852 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50060
> 2010-06-15 12:42:58,860 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=TaskTracker, sessionId=
> 2010-06-15 12:42:58,882 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=TaskTracker, port=50758
> 2010-06-15 12:42:58,960 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2010-06-15 12:42:58,962 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 50758: starting
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 50758: starting
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 50758: starting
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
> TaskTracker up at: localhost.localdomain/127.0.0.1:50758
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting tracker
> tracker_centoshadoop:localhost.localdomain/127.0.0.1:50758
> 2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 50758: starting
> 2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 50758: starting
>
> The config files are:
> core-site.xml:
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/hadoop-0.20.2/tmp/dir/hadoop-${user.name}</value>
> <description> A base for the other temporary
> directories.</description>
> <final>true</final>
> </property>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://10.28.208.118:54310</value>
> <description> The name of the default file system. A URI whose scheme
> and auth
> ority determine the filesystem implementation. The uri's scheme
> determines the c
> onfig property (fs.SCHEME.impl) name the FileSystem implementation
> class. The ur
> i's authority is used to determine the host, port, etc for a
> filesystem</description>
> <final>true</final>
> </property>
>
> </configuration>
>
> hadoop-env.sh
> # Set Hadoop-specific environment variables here.
>
> # The only required environment variable is JAVA_HOME. All others are
> # optional. When running a distributed configuration it is best to
> # set JAVA_HOME in this file, so that it is correctly defined on
> # remote nodes.
>
> # The java implementation to use. Required.
> export JAVA_HOME=/usr/java/default
>
> # The hadoop home directory
> export HADOOP_HOME=/hadoop-0.20.2
>
> # Extra Java CLASSPATH elements. Optional.
> export
>
HADOOP_CLASSPATH=/hbase-0.20.4/lib/zookeeper-3.2.2.jar:/habase-0.20.4/hb
> ase-0.20.4-test.jar:/hbase-0.20.4/hbase-0.20.4.jar:/hbase-0.20.4/conf
>
> # The maximum amount of heap to use, in MB. Default is 1000.
> # export HADOOP_HEAPSIZE=2000
>
> # Extra Java runtime options. Empty by default.
> export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
>
> # Command specific options appended to HADOOP_OPTS when specified
> export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_NAMENODE_OPTS"
> export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_SECONDARYNAMENODE_OPTS"
> export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_DATANODE_OPTS"
> export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_BALANCER_OPTS"
> export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_JOBTRACKER_OPTS"
> # export HADOOP_TASKTRACKER_OPTS=
> # The following applies to multiple commands (fs, dfs, fsck, distcp
etc)
> # export HADOOP_CLIENT_OPTS
>
> # Extra ssh options. Empty by default.
> # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o
> SendEnv=HADOOP_CONF_DIR"
>
> # Where log files are stored. $HADOOP_HOME/logs by default.
> # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
>
> # File naming remote slave hosts. $HADOOP_HOME/conf/slaves by
default.
> # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
>
> # host:path where hadoop code should be rsync'd from. Unset by
default.
> # export HADOOP_MASTER=master:/home/$USER/src/hadoop
>
> # Seconds to sleep between slave commands. Unset by default. This
> # can be useful in large clusters, where, e.g., slave rsyncs can
> # otherwise arrive faster than the master can service them.
> # export HADOOP_SLAVE_SLEEP=0.1
>
> # The directory where pid files are stored. /tmp by default.
> # export HADOOP_PID_DIR=/var/hadoop/pids
>
> # A string representing this instance of hadoop. $USER by default.
> # export HADOOP_IDENT_STRING=$USER
>
> # The scheduling priority for daemon processes. See 'man nice'.
> # export HADOOP_NICENESS=10
>
> hdfs-site.xml
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
> <name>dfs.replication</name>
> <value>22</value>
> <description> Default block replication. The actual number of
> replications can be specified when the file is created. Default is
used
> if not specified at creation time.</description>
> <final>true</final>
> </property>
> <property>
> <name>dfs.data.dir</name>
> <value>/hadoop-0.20.2/hadoopDataStore/data</value>
> <description>DFS Data Directory</description>
> <final>true</final>
> </property>
>
> </configuration>
>
> masters:
> 10.28.208.118
>
> slaves:
> 10.28.208.118
>
> (And because I know it'll be asked)
> hosts:
> # Do not remove the following line, or various programs
> # that require network functionality will fail.
> 127.0.0.1 localhost.localdomain localhost
> ::1 localhost6.localdomain6 localhost6
> 10.28.208.118 centoshadoop centoshadoop.soa.gd-ais.com
>
> Why is this failing to stand up? I'd like to get this one node working
> before I stand up any other data nodes.(Firewall, btw, is turned off)
>
> James Kilbride
Re: IPC connections failing
Posted by Dmytro Molkov <dm...@facebook.com>.
Based on the log of the namenode I do not think it started successfully.
Seems like your edits log is corrupted
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:670)
So all other parts of the system could not connect to the namenode because it was down, not because of IPC problems.
On Jun 15, 2010, at 10:11 AM, Kilbride, James P. wrote:
> Forgot to add on mapred-site.xml:
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
> <name>mapred.job.tracker</name>
> <value>hdfs://10.28.208.118:54311</value>
> <description> The host and port that the MapReduce job tracker runs
> at. If "local", then the jobs are run in-process as a single map and
> reduce task.</description>
> <final>true</final>
> </property>
>
> </configuration>
>
> -----Original Message-----
> From: Kilbride, James P. [mailto:James.Kilbride@gd-ais.com]
> Sent: Tuesday, June 15, 2010 1:10 PM
> To: general@hadoop.apache.org
> Subject: IPC connections failing
>
> I've tried to move my system from pseudo distributed to full
> distributed(with it being the first node in the full cluster I'll be
> setting up) but the components don't seem to be talking to each other
> and I find logs filled with ipc.Client connection failures.
>
> Here's the basic log info:
> DataNode:
> 2010-06-15 12:42:42,824 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:49,335 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:42:50,343 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:42:51,354 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:42:53,641 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:42:54,733 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:42:55,737 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:42:56,824 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:42:57,827 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:42:58,831 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:42:59,833 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:42:59,836 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
> 2010-06-15 12:43:01,841 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:43:02,844 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:43:03,846 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:43:04,849 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:43:05,852 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:06,855 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:07,858 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:08,883 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:09,970 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
>
> JobTracker:
> 2010-06-15 12:42:52,836 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:53,350 INFO org.apache.hadoop.mapred.JobTracker:
> Scheduler configured with (memSizeForMapSlotOnJT,
> memSizeForReduceSlotOnJT, limitMaxMemForMapTasks,
> limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2010-06-15 12:42:53,815 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=JobTracker, port=54311
> 2010-06-15 12:42:53,986 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2010-06-15 12:42:55,381 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50030
> 2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50030
> webServer.getConnectors()[0].getLocalPort() returned 50030
> 2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50030
> 2010-06-15 12:42:55,402 INFO org.mortbay.log: jetty-6.1.14
> 2010-06-15 12:42:56,802 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50030
> 2010-06-15 12:42:56,805 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=JobTracker, sessionId=
> 2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker up at: 54311
> 2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker webserver: 50030
> 2010-06-15 12:42:58,076 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:42:59,080 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:43:00,083 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:43:01,086 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:43:02,089 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:03,091 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:04,095 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:05,174 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:06,177 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:07,180 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:07,185 INFO org.apache.hadoop.mapred.JobTracker:
> problem cleaning system directory: null
> java.net.ConnectException: Call to /10.28.208.118:54310 failed on
> connection exception: java.net.ConnectException: Connection refused
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:767)
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy4.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> at
> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileS
> ystem.java:82)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
> at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
> at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
> at
> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1665)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> at
> org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.ja
> va:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
> at
> org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:860)
> at org.apache.hadoop.ipc.Client.call(Client.java:720)
> ... 16 more
>
> NameNode:
> 2010-06-15 12:42:38,714 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:38,922 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=NameNode, port=54310
> 2010-06-15 12:42:38,932 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at:
> centoshadoop/10.28.208.118:54310
> 2010-06-15 12:42:38,938 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=NameNode, sessionId=null
> 2010-06-15 12:42:38,941 INFO
> org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics:
> Initializing NameNodeMeterics using context
> object:org.apache.hadoop.metrics.spi.NullContext
> 2010-06-15 12:42:39,140 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> fsOwner=hadoop,users,hadoop
> 2010-06-15 12:42:39,141 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> supergroup=supergroup
> 2010-06-15 12:42:39,141 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled=true
> 2010-06-15 12:42:39,156 INFO
> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
> Initializing FSNamesystemMetrics using context
> object:org.apache.hadoop.metrics.spi.NullContext
> 2010-06-15 12:42:39,161 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStatusMBean
> 2010-06-15 12:42:39,381 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Number of files = 28
> 2010-06-15 12:42:39,404 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Number of files under
> construction = 0
> 2010-06-15 12:42:39,404 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Image file of size 2780
> loaded in 0 seconds.
> 2010-06-15 12:42:39,429 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode:
> java.lang.NumberFormatException: For input string: ""
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.jav
> a:48)
> at java.lang.Long.parseLong(Long.java:431)
> at java.lang.Long.parseLong(Long.java:468)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.readLong(FSEditLog.java
> :1273)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
> ava:670)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:
> 992)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:
> 812)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSI
> mage.java:364)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirecto
> ry.java:87)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesys
> tem.java:311)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
> java:292)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java
> :201)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279
> )
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
> java:956)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>
> SecondaryNameNode:
> 2010-06-15 12:42:53,227 INFO
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting SecondaryNameNode
> STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:53,442 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=SecondaryNameNode,
> sessionId=null
> 2010-06-15 12:42:55,651 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:42:56,659 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:42:57,662 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:42:58,665 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:42:59,668 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:00,671 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:01,674 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:02,677 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:03,679 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:04,682 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:04,685 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
> 2010-06-15 12:43:06,691 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:43:07,694 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:43:08,697 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:43:09,700 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:43:10,703 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:11,706 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:12,709 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:13,712 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:14,714 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:15,717 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:15,718 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
>
> TaskTracker:
> 2010-06-15 12:42:57,270 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:57,854 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2010-06-15 12:42:58,328 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50060
> 2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50060
> 2010-06-15 12:42:58,345 INFO org.mortbay.log: jetty-6.1.14
> 2010-06-15 12:42:58,852 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50060
> 2010-06-15 12:42:58,860 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=TaskTracker, sessionId=
> 2010-06-15 12:42:58,882 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=TaskTracker, port=50758
> 2010-06-15 12:42:58,960 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2010-06-15 12:42:58,962 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 50758: starting
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 50758: starting
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 50758: starting
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
> TaskTracker up at: localhost.localdomain/127.0.0.1:50758
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting tracker
> tracker_centoshadoop:localhost.localdomain/127.0.0.1:50758
> 2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 50758: starting
> 2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 50758: starting
>
> The config files are:
> core-site.xml:
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/hadoop-0.20.2/tmp/dir/hadoop-${user.name}</value>
> <description> A base for the other temporary
> directories.</description>
> <final>true</final>
> </property>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://10.28.208.118:54310</value>
> <description> The name of the default file system. A URI whose scheme
> and auth
> ority determine the filesystem implementation. The uri's scheme
> determines the c
> onfig property (fs.SCHEME.impl) name the FileSystem implementation
> class. The ur
> i's authority is used to determine the host, port, etc for a
> filesystem</description>
> <final>true</final>
> </property>
>
> </configuration>
>
> hadoop-env.sh
> # Set Hadoop-specific environment variables here.
>
> # The only required environment variable is JAVA_HOME. All others are
> # optional. When running a distributed configuration it is best to
> # set JAVA_HOME in this file, so that it is correctly defined on
> # remote nodes.
>
> # The java implementation to use. Required.
> export JAVA_HOME=/usr/java/default
>
> # The hadoop home directory
> export HADOOP_HOME=/hadoop-0.20.2
>
> # Extra Java CLASSPATH elements. Optional.
> export
> HADOOP_CLASSPATH=/hbase-0.20.4/lib/zookeeper-3.2.2.jar:/habase-0.20.4/hb
> ase-0.20.4-test.jar:/hbase-0.20.4/hbase-0.20.4.jar:/hbase-0.20.4/conf
>
> # The maximum amount of heap to use, in MB. Default is 1000.
> # export HADOOP_HEAPSIZE=2000
>
> # Extra Java runtime options. Empty by default.
> export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
>
> # Command specific options appended to HADOOP_OPTS when specified
> export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_NAMENODE_OPTS"
> export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_SECONDARYNAMENODE_OPTS"
> export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_DATANODE_OPTS"
> export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_BALANCER_OPTS"
> export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_JOBTRACKER_OPTS"
> # export HADOOP_TASKTRACKER_OPTS=
> # The following applies to multiple commands (fs, dfs, fsck, distcp etc)
> # export HADOOP_CLIENT_OPTS
>
> # Extra ssh options. Empty by default.
> # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o
> SendEnv=HADOOP_CONF_DIR"
>
> # Where log files are stored. $HADOOP_HOME/logs by default.
> # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
>
> # File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
> # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
>
> # host:path where hadoop code should be rsync'd from. Unset by default.
> # export HADOOP_MASTER=master:/home/$USER/src/hadoop
>
> # Seconds to sleep between slave commands. Unset by default. This
> # can be useful in large clusters, where, e.g., slave rsyncs can
> # otherwise arrive faster than the master can service them.
> # export HADOOP_SLAVE_SLEEP=0.1
>
> # The directory where pid files are stored. /tmp by default.
> # export HADOOP_PID_DIR=/var/hadoop/pids
>
> # A string representing this instance of hadoop. $USER by default.
> # export HADOOP_IDENT_STRING=$USER
>
> # The scheduling priority for daemon processes. See 'man nice'.
> # export HADOOP_NICENESS=10
>
> hdfs-site.xml
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
> <name>dfs.replication</name>
> <value>22</value>
> <description> Default block replication. The actual number of
> replications can be specified when the file is created. Default is used
> if not specified at creation time.</description>
> <final>true</final>
> </property>
> <property>
> <name>dfs.data.dir</name>
> <value>/hadoop-0.20.2/hadoopDataStore/data</value>
> <description>DFS Data Directory</description>
> <final>true</final>
> </property>
>
> </configuration>
>
> masters:
> 10.28.208.118
>
> slaves:
> 10.28.208.118
>
> (And because I know it'll be asked)
> hosts:
> # Do not remove the following line, or various programs
> # that require network functionality will fail.
> 127.0.0.1 localhost.localdomain localhost
> ::1 localhost6.localdomain6 localhost6
> 10.28.208.118 centoshadoop centoshadoop.soa.gd-ais.com
>
> Why is this failing to stand up? I'd like to get this one node working
> before I stand up any other data nodes.(Firewall, btw, is turned off)
>
> James Kilbride
RE: IPC connections failing
Posted by "Kilbride, James P." <Ja...@gd-ais.com>.
Forgot to add on mapred-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://10.28.208.118:54311</value>
<description> The host and port that the MapReduce job tracker runs
at. If "local", then the jobs are run in-process as a single map and
reduce task.</description>
<final>true</final>
</property>
</configuration>
-----Original Message-----
From: Kilbride, James P. [mailto:James.Kilbride@gd-ais.com]
Sent: Tuesday, June 15, 2010 1:10 PM
To: general@hadoop.apache.org
Subject: IPC connections failing
I've tried to move my system from pseudo distributed to full
distributed(with it being the first node in the full cluster I'll be
setting up) but the components don't seem to be talking to each other
and I find logs filled with ipc.Client connection failures.
Here's the basic log info:
DataNode:
2010-06-15 12:42:42,824 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:49,335 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:42:50,343 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:42:51,354 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:42:53,641 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:42:54,733 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:42:55,737 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:42:56,824 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:42:57,827 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:42:58,831 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:42:59,833 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:42:59,836 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...
2010-06-15 12:43:01,841 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:43:02,844 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:43:03,846 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:43:04,849 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:43:05,852 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:06,855 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:07,858 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:08,883 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:09,970 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...
JobTracker:
2010-06-15 12:42:52,836 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:53,350 INFO org.apache.hadoop.mapred.JobTracker:
Scheduler configured with (memSizeForMapSlotOnJT,
memSizeForReduceSlotOnJT, limitMaxMemForMapTasks,
limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2010-06-15 12:42:53,815 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=JobTracker, port=54311
2010-06-15 12:42:53,986 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2010-06-15 12:42:55,381 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is
-1. Opening the listener on 50030
2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50030
webServer.getConnectors()[0].getLocalPort() returned 50030
2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer: Jetty
bound to port 50030
2010-06-15 12:42:55,402 INFO org.mortbay.log: jetty-6.1.14
2010-06-15 12:42:56,802 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50030
2010-06-15 12:42:56,805 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=JobTracker, sessionId=
2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker up at: 54311
2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker webserver: 50030
2010-06-15 12:42:58,076 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:42:59,080 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:43:00,083 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:43:01,086 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:43:02,089 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:03,091 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:04,095 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:05,174 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:06,177 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:07,180 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:07,185 INFO org.apache.hadoop.mapred.JobTracker:
problem cleaning system directory: null
java.net.ConnectException: Call to /10.28.208.118:54310 failed on
connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:767)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileS
ystem.java:82)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at
org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1665)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
at
org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.ja
va:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
at
org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:860)
at org.apache.hadoop.ipc.Client.call(Client.java:720)
... 16 more
NameNode:
2010-06-15 12:42:38,714 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:38,922 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=NameNode, port=54310
2010-06-15 12:42:38,932 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at:
centoshadoop/10.28.208.118:54310
2010-06-15 12:42:38,938 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=NameNode, sessionId=null
2010-06-15 12:42:38,941 INFO
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics:
Initializing NameNodeMeterics using context
object:org.apache.hadoop.metrics.spi.NullContext
2010-06-15 12:42:39,140 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
fsOwner=hadoop,users,hadoop
2010-06-15 12:42:39,141 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
supergroup=supergroup
2010-06-15 12:42:39,141 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
isPermissionEnabled=true
2010-06-15 12:42:39,156 INFO
org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
Initializing FSNamesystemMetrics using context
object:org.apache.hadoop.metrics.spi.NullContext
2010-06-15 12:42:39,161 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean
2010-06-15 12:42:39,381 INFO
org.apache.hadoop.hdfs.server.common.Storage: Number of files = 28
2010-06-15 12:42:39,404 INFO
org.apache.hadoop.hdfs.server.common.Storage: Number of files under
construction = 0
2010-06-15 12:42:39,404 INFO
org.apache.hadoop.hdfs.server.common.Storage: Image file of size 2780
loaded in 0 seconds.
2010-06-15 12:42:39,429 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:
java.lang.NumberFormatException: For input string: ""
at
java.lang.NumberFormatException.forInputString(NumberFormatException.jav
a:48)
at java.lang.Long.parseLong(Long.java:431)
at java.lang.Long.parseLong(Long.java:468)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.readLong(FSEditLog.java
:1273)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
ava:670)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:
992)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:
812)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSI
mage.java:364)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirecto
ry.java:87)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesys
tem.java:311)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
java:292)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java
:201)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279
)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
java:956)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
SecondaryNameNode:
2010-06-15 12:42:53,227 INFO
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:53,442 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=SecondaryNameNode,
sessionId=null
2010-06-15 12:42:55,651 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:42:56,659 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:42:57,662 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:42:58,665 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:42:59,668 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:00,671 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:01,674 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:02,677 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:03,679 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:04,682 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:04,685 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...
2010-06-15 12:43:06,691 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:43:07,694 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:43:08,697 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:43:09,700 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:43:10,703 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:11,706 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:12,709 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:13,712 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:14,714 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:15,717 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:15,718 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...
TaskTracker:
2010-06-15 12:42:57,270 INFO org.apache.hadoop.mapred.TaskTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:57,854 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2010-06-15 12:42:58,328 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is
-1. Opening the listener on 50060
2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50060
webServer.getConnectors()[0].getLocalPort() returned 50060
2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer: Jetty
bound to port 50060
2010-06-15 12:42:58,345 INFO org.mortbay.log: jetty-6.1.14
2010-06-15 12:42:58,852 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50060
2010-06-15 12:42:58,860 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=TaskTracker, sessionId=
2010-06-15 12:42:58,882 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=TaskTracker, port=50758
2010-06-15 12:42:58,960 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2010-06-15 12:42:58,962 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 50758: starting
2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 50758: starting
2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 50758: starting
2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
TaskTracker up at: localhost.localdomain/127.0.0.1:50758
2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
Starting tracker
tracker_centoshadoop:localhost.localdomain/127.0.0.1:50758
2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 50758: starting
2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 50758: starting
The config files are:
core-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop-0.20.2/tmp/dir/hadoop-${user.name}</value>
<description> A base for the other temporary
directories.</description>
<final>true</final>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://10.28.208.118:54310</value>
<description> The name of the default file system. A URI whose scheme
and auth
ority determine the filesystem implementation. The uri's scheme
determines the c
onfig property (fs.SCHEME.impl) name the FileSystem implementation
class. The ur
i's authority is used to determine the host, port, etc for a
filesystem</description>
<final>true</final>
</property>
</configuration>
hadoop-env.sh
# Set Hadoop-specific environment variables here.
# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.
# The java implementation to use. Required.
export JAVA_HOME=/usr/java/default
# The hadoop home directory
export HADOOP_HOME=/hadoop-0.20.2
# Extra Java CLASSPATH elements. Optional.
export
HADOOP_CLASSPATH=/hbase-0.20.4/lib/zookeeper-3.2.2.jar:/habase-0.20.4/hb
ase-0.20.4-test.jar:/hbase-0.20.4/hbase-0.20.4.jar:/hbase-0.20.4/conf
# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000
# Extra Java runtime options. Empty by default.
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS
# Extra ssh options. Empty by default.
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o
SendEnv=HADOOP_CONF_DIR"
# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
# File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
# host:path where hadoop code should be rsync'd from. Unset by default.
# export HADOOP_MASTER=master:/home/$USER/src/hadoop
# Seconds to sleep between slave commands. Unset by default. This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=0.1
# The directory where pid files are stored. /tmp by default.
# export HADOOP_PID_DIR=/var/hadoop/pids
# A string representing this instance of hadoop. $USER by default.
# export HADOOP_IDENT_STRING=$USER
# The scheduling priority for daemon processes. See 'man nice'.
# export HADOOP_NICENESS=10
hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>22</value>
<description> Default block replication. The actual number of
replications can be specified when the file is created. Default is used
if not specified at creation time.</description>
<final>true</final>
</property>
<property>
<name>dfs.data.dir</name>
<value>/hadoop-0.20.2/hadoopDataStore/data</value>
<description>DFS Data Directory</description>
<final>true</final>
</property>
</configuration>
masters:
10.28.208.118
slaves:
10.28.208.118
(And because I know it'll be asked)
hosts:
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
10.28.208.118 centoshadoop centoshadoop.soa.gd-ais.com
Why is this failing to stand up? I'd like to get this one node working
before I stand up any other data nodes.(Firewall, btw, is turned off)
James Kilbride