You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by bikash sharma <sh...@gmail.com> on 2011/04/13 15:20:31 UTC
hadoop cluster installation problems
Hi,
I need to install hadoop on 16-node cluster. I have a couple of related
questions:
1. I have installed hadoop on a shared directory, i.e., there is just one
place where the whole hadoop installation files exist and all the 16 nodes
use the same installation.
Is that an issue or I need to install hadoop on each of these nodes in their
local directory separately?
2. I installed hadoop-0.21 and after following the installation
instructions, when i tried formatting, I get the following error:
************************************************************/
Re-format filesystem in /var/tmp/data/dfs/name ? (Y or N) Y
11/04/13 09:16:23 INFO namenode.FSNamesystem: defaultReplication = 3
11/04/13 09:16:23 INFO namenode.FSNamesystem: maxReplication = 512
11/04/13 09:16:23 INFO namenode.FSNamesystem: minReplication = 1
11/04/13 09:16:23 INFO namenode.FSNamesystem: maxReplicationStreams = 2
11/04/13 09:16:23 INFO namenode.FSNamesystem: shouldCheckForEnoughRacks =
false
11/04/13 09:16:23 INFO security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
cacheTimeout=300000
11/04/13 09:16:23 INFO namenode.FSNamesystem: fsOwner=bus145
11/04/13 09:16:23 INFO namenode.FSNamesystem: supergroup=supergroup
11/04/13 09:16:23 INFO namenode.FSNamesystem: isPermissionEnabled=true
11/04/13 09:16:23 INFO namenode.FSNamesystem: isAccessTokenEnabled=false
accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
11/04/13 09:16:24 INFO common.Storage: Cannot lock storage
/var/tmp/data/dfs/name. The directory is already locked.
11/04/13 09:16:24 ERROR namenode.NameNode: java.io.IOException: Cannot lock
storage /var/tmp/data/dfs/name. The directory is already locked.
at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:617)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1426)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1444)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1242)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
11/04/13 09:16:24 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at inti79.cse.psu.edu/130.203.58.207
************************************************************/
3. I was using before hadoop-0.20, and formatting was working fine.
4. Also, when i do a bin/start-dfs.sh, am able to see the Namenode, Datanode
up, however on bin/start-mapred.sh, am not able to see Jobtracker up on the
master node, though Tasktracker seems to be up on slaves.
Before upgrading to Hadoop-0.21, everything was working fine with
hadoop-0.20 including running benchmarks and getting stats.
Any suggestions in this regard is highly appreciated.
Thanks,
Bikash
Re: hadoop cluster installation problems
Posted by bikash sharma <sh...@gmail.com>.
p.s.
Also, while starting dfs using bin/start-dfs.sh, I get the following error:
2011-04-13 09:42:31,729 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = inti84.cse.psu.edu/130.203.58.212
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2011-04-13 09:42:31,853 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:
java.lang.NullPointerException
at
org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:156)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:160)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:175)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
2011-04-13 09:42:31,854 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at inti84.cse.psu.edu/130.203.58.212
************************************************************/
2011-04-13 09:44:03,265 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = inti84.cse.psu.edu/130.203.58.212
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2011-04-13 09:44:03,384 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:
java.lang.NullPointerException
at
org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:156)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:160)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:175)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
On Wed, Apr 13, 2011 at 9:20 AM, bikash sharma <sh...@gmail.com>wrote:
> Hi,
> I need to install hadoop on 16-node cluster. I have a couple of related
> questions:
> 1. I have installed hadoop on a shared directory, i.e., there is just one
> place where the whole hadoop installation files exist and all the 16 nodes
> use the same installation.
> Is that an issue or I need to install hadoop on each of these nodes in
> their local directory separately?
> 2. I installed hadoop-0.21 and after following the installation
> instructions, when i tried formatting, I get the following error:
>
> ************************************************************/
> Re-format filesystem in /var/tmp/data/dfs/name ? (Y or N) Y
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: defaultReplication = 3
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: maxReplication = 512
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: minReplication = 1
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: maxReplicationStreams = 2
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: shouldCheckForEnoughRacks =
> false
> 11/04/13 09:16:23 INFO security.Groups: Group mapping
> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
> cacheTimeout=300000
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: fsOwner=bus145
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: supergroup=supergroup
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: isPermissionEnabled=true
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: isAccessTokenEnabled=false
> accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
> 11/04/13 09:16:24 INFO common.Storage: Cannot lock storage
> /var/tmp/data/dfs/name. The directory is already locked.
> 11/04/13 09:16:24 ERROR namenode.NameNode: java.io.IOException: Cannot lock
> storage /var/tmp/data/dfs/name. The directory is already locked.
> at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:617)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1426)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1444)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1242)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
>
> 11/04/13 09:16:24 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at inti79.cse.psu.edu/130.203.58.207
> ************************************************************/
>
> 3. I was using before hadoop-0.20, and formatting was working fine.
> 4. Also, when i do a bin/start-dfs.sh, am able to see the Namenode,
> Datanode up, however on bin/start-mapred.sh, am not able to see Jobtracker
> up on the master node, though Tasktracker seems to be up on slaves.
>
> Before upgrading to Hadoop-0.21, everything was working fine with
> hadoop-0.20 including running benchmarks and getting stats.
>
> Any suggestions in this regard is highly appreciated.
>
> Thanks,
> Bikash
>
Re: hadoop cluster installation problems
Posted by bikash sharma <sh...@gmail.com>.
p.s.
Also, while starting dfs using bin/start-dfs.sh, I get the following error:
2011-04-13 09:42:31,729 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = inti84.cse.psu.edu/130.203.58.212
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2011-04-13 09:42:31,853 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:
java.lang.NullPointerException
at
org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:156)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:160)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:175)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
2011-04-13 09:42:31,854 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at inti84.cse.psu.edu/130.203.58.212
************************************************************/
2011-04-13 09:44:03,265 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = inti84.cse.psu.edu/130.203.58.212
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2011-04-13 09:44:03,384 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:
java.lang.NullPointerException
at
org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:156)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:160)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:175)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
On Wed, Apr 13, 2011 at 9:20 AM, bikash sharma <sh...@gmail.com>wrote:
> Hi,
> I need to install hadoop on 16-node cluster. I have a couple of related
> questions:
> 1. I have installed hadoop on a shared directory, i.e., there is just one
> place where the whole hadoop installation files exist and all the 16 nodes
> use the same installation.
> Is that an issue or I need to install hadoop on each of these nodes in
> their local directory separately?
> 2. I installed hadoop-0.21 and after following the installation
> instructions, when i tried formatting, I get the following error:
>
> ************************************************************/
> Re-format filesystem in /var/tmp/data/dfs/name ? (Y or N) Y
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: defaultReplication = 3
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: maxReplication = 512
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: minReplication = 1
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: maxReplicationStreams = 2
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: shouldCheckForEnoughRacks =
> false
> 11/04/13 09:16:23 INFO security.Groups: Group mapping
> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
> cacheTimeout=300000
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: fsOwner=bus145
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: supergroup=supergroup
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: isPermissionEnabled=true
> 11/04/13 09:16:23 INFO namenode.FSNamesystem: isAccessTokenEnabled=false
> accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
> 11/04/13 09:16:24 INFO common.Storage: Cannot lock storage
> /var/tmp/data/dfs/name. The directory is already locked.
> 11/04/13 09:16:24 ERROR namenode.NameNode: java.io.IOException: Cannot lock
> storage /var/tmp/data/dfs/name. The directory is already locked.
> at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:617)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1426)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1444)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1242)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
>
> 11/04/13 09:16:24 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at inti79.cse.psu.edu/130.203.58.207
> ************************************************************/
>
> 3. I was using before hadoop-0.20, and formatting was working fine.
> 4. Also, when i do a bin/start-dfs.sh, am able to see the Namenode,
> Datanode up, however on bin/start-mapred.sh, am not able to see Jobtracker
> up on the master node, though Tasktracker seems to be up on slaves.
>
> Before upgrading to Hadoop-0.21, everything was working fine with
> hadoop-0.20 including running benchmarks and getting stats.
>
> Any suggestions in this regard is highly appreciated.
>
> Thanks,
> Bikash
>