You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Mohit Vadhera <pr...@gmail.com> on 2013/02/28 08:58:48 UTC
namenode is failing
Hi Guys,
Namenode switches into safemode when it has low disk space on the root fs /
i have to manually run a command to leave it
I have space on other partition. Can I change the path for cache files on
other partition ? I have below properties . Can it resolve the issue ? If i
change the path to other directories and restart services I get the below
error while starting the service namenode. I didn't find anything in logs
so far. Can you please suggest something ?
<property>
<name>hadoop.tmp.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name
}/dfs/namesecondary</value>
</property>
<property>
Service namenode is failing
# for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
Hadoop datanode is running [ OK ]
Hadoop namenode is dead and pid file exists [FAILED]
Hadoop secondarynamenode is running [ OK ]
Thanks,
Re: namenode is failing
Posted by Harsh J <ha...@cloudera.com>.
If you move the NN dir, you also have to move its older location's
contents there before starting the NN.
On Thu, Feb 28, 2013 at 1:28 PM, Mohit Vadhera
<pr...@gmail.com> wrote:
> Hi Guys,
>
> Namenode switches into safemode when it has low disk space on the root fs /
> i have to manually run a command to leave it
> I have space on other partition. Can I change the path for cache files on
> other partition ? I have below properties . Can it resolve the issue ? If i
> change the path to other directories and restart services I get the below
> error while starting the service namenode. I didn't find anything in logs so
> far. Can you please suggest something ?
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> </property>
> <property>
> <name>dfs.namenode.name.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
> </property>
> <property>
> <name>dfs.namenode.checkpoint.dir</name>
>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/namesecondary</value>
> </property>
> <property>
>
>
> Service namenode is failing
>
> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
> Hadoop datanode is running [ OK ]
> Hadoop namenode is dead and pid file exists [FAILED]
> Hadoop secondarynamenode is running [ OK ]
>
> Thanks,
>
--
Harsh J
Re: namenode is failing
Posted by YouPeng Yang <yy...@gmail.com>.
Hi* *Mohit
Your fsOwner hdfs should have the permission to access to the
/mnt/san1/hdfs/cache/hdfs/dfs/name.
So please check the permission of /mnt/ , and sub-directories on the OS.
they all need to be read ,writen.
regards
2013/2/28 Mohit Vadhera <pr...@gmail.com>
> Please find below logs for shutting down the namenode service. Can anybody
> check this
>
> 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 02:07:51,754 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
> 2013-02-28 02:07:51,754 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
> 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-02-28 02:07:51,890 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
> 2013-02-28 02:07:51,909 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
> = 512
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
> = 1
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> hdfs (auth:SIMPLE)
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> hadmin
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> 2013-02-28 02:07:51,925 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> 2013-02-28 02:07:52,462 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
> 2013-02-28 02:07:52,466 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> 2013-02-28 02:07:52,467 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
> 2013-02-28 02:07:52,467 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 0
> 2013-02-28 02:07:52,469 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
> 2013-02-28 02:07:52,471 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
> 2013-02-28 02:07:52,472 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 02:07:52,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 02:07:52,473 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
> storage directory does not exist or is not accessible.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2013-02-28 02:08:48,913 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
> 192.168.1.3
>
>
> On Thu, Feb 28, 2013 at 1:28 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> Hi Guys,
>>
>> Namenode switches into safemode when it has low disk space on the root fs
>> / i have to manually run a command to leave it
>> I have space on other partition. Can I change the path for cache files on
>> other partition ? I have below properties . Can it resolve the issue ? If i
>> change the path to other directories and restart services I get the below
>> error while starting the service namenode. I didn't find anything in logs
>> so far. Can you please suggest something ?
>>
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>> </property>
>> <property>
>> <name>dfs.namenode.name.dir</name>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>> </property>
>> <property>
>> <name>dfs.namenode.checkpoint.dir</name>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/namesecondary</value>
>> </property>
>> <property>
>>
>>
>> Service namenode is failing
>>
>> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
>> Hadoop datanode is running [ OK ]
>> Hadoop namenode is dead and pid file exists [FAILED]
>> Hadoop secondarynamenode is running [ OK ]
>>
>> Thanks,
>>
>>
>
Re: namenode is failing
Posted by YouPeng Yang <yy...@gmail.com>.
Hi* *Mohit
Your fsOwner hdfs should have the permission to access to the
/mnt/san1/hdfs/cache/hdfs/dfs/name.
So please check the permission of /mnt/ , and sub-directories on the OS.
they all need to be read ,writen.
regards
2013/2/28 Mohit Vadhera <pr...@gmail.com>
> Please find below logs for shutting down the namenode service. Can anybody
> check this
>
> 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 02:07:51,754 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
> 2013-02-28 02:07:51,754 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
> 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-02-28 02:07:51,890 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
> 2013-02-28 02:07:51,909 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
> = 512
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
> = 1
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> hdfs (auth:SIMPLE)
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> hadmin
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> 2013-02-28 02:07:51,925 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> 2013-02-28 02:07:52,462 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
> 2013-02-28 02:07:52,466 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> 2013-02-28 02:07:52,467 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
> 2013-02-28 02:07:52,467 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 0
> 2013-02-28 02:07:52,469 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
> 2013-02-28 02:07:52,471 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
> 2013-02-28 02:07:52,472 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 02:07:52,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 02:07:52,473 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
> storage directory does not exist or is not accessible.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2013-02-28 02:08:48,913 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
> 192.168.1.3
>
>
> On Thu, Feb 28, 2013 at 1:28 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> Hi Guys,
>>
>> Namenode switches into safemode when it has low disk space on the root fs
>> / i have to manually run a command to leave it
>> I have space on other partition. Can I change the path for cache files on
>> other partition ? I have below properties . Can it resolve the issue ? If i
>> change the path to other directories and restart services I get the below
>> error while starting the service namenode. I didn't find anything in logs
>> so far. Can you please suggest something ?
>>
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>> </property>
>> <property>
>> <name>dfs.namenode.name.dir</name>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>> </property>
>> <property>
>> <name>dfs.namenode.checkpoint.dir</name>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/namesecondary</value>
>> </property>
>> <property>
>>
>>
>> Service namenode is failing
>>
>> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
>> Hadoop datanode is running [ OK ]
>> Hadoop namenode is dead and pid file exists [FAILED]
>> Hadoop secondarynamenode is running [ OK ]
>>
>> Thanks,
>>
>>
>
Re: namenode is failing
Posted by YouPeng Yang <yy...@gmail.com>.
Hi* *Mohit
Your fsOwner hdfs should have the permission to access to the
/mnt/san1/hdfs/cache/hdfs/dfs/name.
So please check the permission of /mnt/ , and sub-directories on the OS.
they all need to be read ,writen.
regards
2013/2/28 Mohit Vadhera <pr...@gmail.com>
> Please find below logs for shutting down the namenode service. Can anybody
> check this
>
> 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 02:07:51,754 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
> 2013-02-28 02:07:51,754 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
> 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-02-28 02:07:51,890 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
> 2013-02-28 02:07:51,909 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
> = 512
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
> = 1
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> hdfs (auth:SIMPLE)
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> hadmin
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> 2013-02-28 02:07:51,925 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> 2013-02-28 02:07:52,462 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
> 2013-02-28 02:07:52,466 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> 2013-02-28 02:07:52,467 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
> 2013-02-28 02:07:52,467 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 0
> 2013-02-28 02:07:52,469 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
> 2013-02-28 02:07:52,471 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
> 2013-02-28 02:07:52,472 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 02:07:52,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 02:07:52,473 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
> storage directory does not exist or is not accessible.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2013-02-28 02:08:48,913 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
> 192.168.1.3
>
>
> On Thu, Feb 28, 2013 at 1:28 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> Hi Guys,
>>
>> Namenode switches into safemode when it has low disk space on the root fs
>> / i have to manually run a command to leave it
>> I have space on other partition. Can I change the path for cache files on
>> other partition ? I have below properties . Can it resolve the issue ? If i
>> change the path to other directories and restart services I get the below
>> error while starting the service namenode. I didn't find anything in logs
>> so far. Can you please suggest something ?
>>
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>> </property>
>> <property>
>> <name>dfs.namenode.name.dir</name>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>> </property>
>> <property>
>> <name>dfs.namenode.checkpoint.dir</name>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/namesecondary</value>
>> </property>
>> <property>
>>
>>
>> Service namenode is failing
>>
>> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
>> Hadoop datanode is running [ OK ]
>> Hadoop namenode is dead and pid file exists [FAILED]
>> Hadoop secondarynamenode is running [ OK ]
>>
>> Thanks,
>>
>>
>
Re: namenode is failing
Posted by YouPeng Yang <yy...@gmail.com>.
Hi* *Mohit
Your fsOwner hdfs should have the permission to access to the
/mnt/san1/hdfs/cache/hdfs/dfs/name.
So please check the permission of /mnt/ , and sub-directories on the OS.
they all need to be read ,writen.
regards
2013/2/28 Mohit Vadhera <pr...@gmail.com>
> Please find below logs for shutting down the namenode service. Can anybody
> check this
>
> 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 02:07:51,754 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
> 2013-02-28 02:07:51,754 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
> 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-02-28 02:07:51,890 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
> 2013-02-28 02:07:51,909 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
> = 512
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
> = 1
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> hdfs (auth:SIMPLE)
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> hadmin
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> 2013-02-28 02:07:51,925 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> 2013-02-28 02:07:52,462 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
> 2013-02-28 02:07:52,466 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> 2013-02-28 02:07:52,467 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
> 2013-02-28 02:07:52,467 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 0
> 2013-02-28 02:07:52,469 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
> 2013-02-28 02:07:52,471 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
> 2013-02-28 02:07:52,472 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 02:07:52,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 02:07:52,473 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
> storage directory does not exist or is not accessible.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2013-02-28 02:08:48,913 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
> 192.168.1.3
>
>
> On Thu, Feb 28, 2013 at 1:28 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> Hi Guys,
>>
>> Namenode switches into safemode when it has low disk space on the root fs
>> / i have to manually run a command to leave it
>> I have space on other partition. Can I change the path for cache files on
>> other partition ? I have below properties . Can it resolve the issue ? If i
>> change the path to other directories and restart services I get the below
>> error while starting the service namenode. I didn't find anything in logs
>> so far. Can you please suggest something ?
>>
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>> </property>
>> <property>
>> <name>dfs.namenode.name.dir</name>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>> </property>
>> <property>
>> <name>dfs.namenode.checkpoint.dir</name>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/namesecondary</value>
>> </property>
>> <property>
>>
>>
>> Service namenode is failing
>>
>> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
>> Hadoop datanode is running [ OK ]
>> Hadoop namenode is dead and pid file exists [FAILED]
>> Hadoop secondarynamenode is running [ OK ]
>>
>> Thanks,
>>
>>
>
Re: namenode is failing
Posted by Mohit Vadhera <pr...@gmail.com>.
Please find below logs for shutting down the namenode service. Can anybody
check this
2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant storage directories!
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of redundant storage directories!
2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2013-02-28 02:07:51,890 INFO
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=1000
2013-02-28 02:07:51,909 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.block.access.token.enable=false
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
defaultReplication = 1
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxReplicationStreams = 2
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
shouldCheckForEnoughRacks = false
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
replicationRecheckInterval = 3000
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
encryptDataTransfer = false
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
hdfs (auth:SIMPLE)
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
hadmin
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 02:07:51,925 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 02:07:52,462 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2013-02-28 02:07:52,466 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 02:07:52,467 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 02:07:52,467 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.extension = 0
2013-02-28 02:07:52,469 INFO org.apache.hadoop.hdfs.server.common.Storage:
Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
2013-02-28 02:07:52,471 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 02:07:52,472 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 02:07:52,473 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 02:07:52,473 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
storage directory does not exist or is not accessible.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2013-02-28 02:08:48,913 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/192.168.1.3
On Thu, Feb 28, 2013 at 1:28 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> Hi Guys,
>
> Namenode switches into safemode when it has low disk space on the root fs
> / i have to manually run a command to leave it
> I have space on other partition. Can I change the path for cache files on
> other partition ? I have below properties . Can it resolve the issue ? If i
> change the path to other directories and restart services I get the below
> error while starting the service namenode. I didn't find anything in logs
> so far. Can you please suggest something ?
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> </property>
> <property>
> <name>dfs.namenode.name.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
> </property>
> <property>
> <name>dfs.namenode.checkpoint.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/namesecondary</value>
> </property>
> <property>
>
>
> Service namenode is failing
>
> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
> Hadoop datanode is running [ OK ]
> Hadoop namenode is dead and pid file exists [FAILED]
> Hadoop secondarynamenode is running [ OK ]
>
> Thanks,
>
>
Re: namenode is failing
Posted by Harsh J <ha...@cloudera.com>.
If you move the NN dir, you also have to move its older location's
contents there before starting the NN.
On Thu, Feb 28, 2013 at 1:28 PM, Mohit Vadhera
<pr...@gmail.com> wrote:
> Hi Guys,
>
> Namenode switches into safemode when it has low disk space on the root fs /
> i have to manually run a command to leave it
> I have space on other partition. Can I change the path for cache files on
> other partition ? I have below properties . Can it resolve the issue ? If i
> change the path to other directories and restart services I get the below
> error while starting the service namenode. I didn't find anything in logs so
> far. Can you please suggest something ?
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> </property>
> <property>
> <name>dfs.namenode.name.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
> </property>
> <property>
> <name>dfs.namenode.checkpoint.dir</name>
>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/namesecondary</value>
> </property>
> <property>
>
>
> Service namenode is failing
>
> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
> Hadoop datanode is running [ OK ]
> Hadoop namenode is dead and pid file exists [FAILED]
> Hadoop secondarynamenode is running [ OK ]
>
> Thanks,
>
--
Harsh J
Re: namenode is failing
Posted by Harsh J <ha...@cloudera.com>.
If you move the NN dir, you also have to move its older location's
contents there before starting the NN.
On Thu, Feb 28, 2013 at 1:28 PM, Mohit Vadhera
<pr...@gmail.com> wrote:
> Hi Guys,
>
> Namenode switches into safemode when it has low disk space on the root fs /
> i have to manually run a command to leave it
> I have space on other partition. Can I change the path for cache files on
> other partition ? I have below properties . Can it resolve the issue ? If i
> change the path to other directories and restart services I get the below
> error while starting the service namenode. I didn't find anything in logs so
> far. Can you please suggest something ?
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> </property>
> <property>
> <name>dfs.namenode.name.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
> </property>
> <property>
> <name>dfs.namenode.checkpoint.dir</name>
>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/namesecondary</value>
> </property>
> <property>
>
>
> Service namenode is failing
>
> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
> Hadoop datanode is running [ OK ]
> Hadoop namenode is dead and pid file exists [FAILED]
> Hadoop secondarynamenode is running [ OK ]
>
> Thanks,
>
--
Harsh J
Re: namenode is failing
Posted by Mohit Vadhera <pr...@gmail.com>.
Please find below logs for shutting down the namenode service. Can anybody
check this
2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant storage directories!
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of redundant storage directories!
2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2013-02-28 02:07:51,890 INFO
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=1000
2013-02-28 02:07:51,909 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.block.access.token.enable=false
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
defaultReplication = 1
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxReplicationStreams = 2
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
shouldCheckForEnoughRacks = false
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
replicationRecheckInterval = 3000
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
encryptDataTransfer = false
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
hdfs (auth:SIMPLE)
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
hadmin
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 02:07:51,925 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 02:07:52,462 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2013-02-28 02:07:52,466 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 02:07:52,467 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 02:07:52,467 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.extension = 0
2013-02-28 02:07:52,469 INFO org.apache.hadoop.hdfs.server.common.Storage:
Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
2013-02-28 02:07:52,471 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 02:07:52,472 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 02:07:52,473 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 02:07:52,473 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
storage directory does not exist or is not accessible.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2013-02-28 02:08:48,913 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/192.168.1.3
On Thu, Feb 28, 2013 at 1:28 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> Hi Guys,
>
> Namenode switches into safemode when it has low disk space on the root fs
> / i have to manually run a command to leave it
> I have space on other partition. Can I change the path for cache files on
> other partition ? I have below properties . Can it resolve the issue ? If i
> change the path to other directories and restart services I get the below
> error while starting the service namenode. I didn't find anything in logs
> so far. Can you please suggest something ?
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> </property>
> <property>
> <name>dfs.namenode.name.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
> </property>
> <property>
> <name>dfs.namenode.checkpoint.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/namesecondary</value>
> </property>
> <property>
>
>
> Service namenode is failing
>
> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
> Hadoop datanode is running [ OK ]
> Hadoop namenode is dead and pid file exists [FAILED]
> Hadoop secondarynamenode is running [ OK ]
>
> Thanks,
>
>
Re: namenode is failing
Posted by Mohit Vadhera <pr...@gmail.com>.
Please find below logs for shutting down the namenode service. Can anybody
check this
2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant storage directories!
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of redundant storage directories!
2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2013-02-28 02:07:51,890 INFO
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=1000
2013-02-28 02:07:51,909 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.block.access.token.enable=false
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
defaultReplication = 1
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxReplicationStreams = 2
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
shouldCheckForEnoughRacks = false
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
replicationRecheckInterval = 3000
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
encryptDataTransfer = false
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
hdfs (auth:SIMPLE)
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
hadmin
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 02:07:51,925 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 02:07:52,462 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2013-02-28 02:07:52,466 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 02:07:52,467 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 02:07:52,467 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.extension = 0
2013-02-28 02:07:52,469 INFO org.apache.hadoop.hdfs.server.common.Storage:
Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
2013-02-28 02:07:52,471 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 02:07:52,472 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 02:07:52,473 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 02:07:52,473 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
storage directory does not exist or is not accessible.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2013-02-28 02:08:48,913 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/192.168.1.3
On Thu, Feb 28, 2013 at 1:28 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> Hi Guys,
>
> Namenode switches into safemode when it has low disk space on the root fs
> / i have to manually run a command to leave it
> I have space on other partition. Can I change the path for cache files on
> other partition ? I have below properties . Can it resolve the issue ? If i
> change the path to other directories and restart services I get the below
> error while starting the service namenode. I didn't find anything in logs
> so far. Can you please suggest something ?
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> </property>
> <property>
> <name>dfs.namenode.name.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
> </property>
> <property>
> <name>dfs.namenode.checkpoint.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/namesecondary</value>
> </property>
> <property>
>
>
> Service namenode is failing
>
> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
> Hadoop datanode is running [ OK ]
> Hadoop namenode is dead and pid file exists [FAILED]
> Hadoop secondarynamenode is running [ OK ]
>
> Thanks,
>
>
Re: namenode is failing
Posted by Mohit Vadhera <pr...@gmail.com>.
Please find below logs for shutting down the namenode service. Can anybody
check this
2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant storage directories!
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of redundant storage directories!
2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2013-02-28 02:07:51,890 INFO
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=1000
2013-02-28 02:07:51,909 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.block.access.token.enable=false
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
defaultReplication = 1
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxReplicationStreams = 2
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
shouldCheckForEnoughRacks = false
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
replicationRecheckInterval = 3000
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
encryptDataTransfer = false
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
hdfs (auth:SIMPLE)
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
hadmin
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 02:07:51,925 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 02:07:52,462 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2013-02-28 02:07:52,466 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 02:07:52,467 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 02:07:52,467 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.extension = 0
2013-02-28 02:07:52,469 INFO org.apache.hadoop.hdfs.server.common.Storage:
Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
2013-02-28 02:07:52,471 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 02:07:52,472 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 02:07:52,473 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 02:07:52,473 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
storage directory does not exist or is not accessible.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2013-02-28 02:08:48,913 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/192.168.1.3
On Thu, Feb 28, 2013 at 1:28 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> Hi Guys,
>
> Namenode switches into safemode when it has low disk space on the root fs
> / i have to manually run a command to leave it
> I have space on other partition. Can I change the path for cache files on
> other partition ? I have below properties . Can it resolve the issue ? If i
> change the path to other directories and restart services I get the below
> error while starting the service namenode. I didn't find anything in logs
> so far. Can you please suggest something ?
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> </property>
> <property>
> <name>dfs.namenode.name.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
> </property>
> <property>
> <name>dfs.namenode.checkpoint.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/namesecondary</value>
> </property>
> <property>
>
>
> Service namenode is failing
>
> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
> Hadoop datanode is running [ OK ]
> Hadoop namenode is dead and pid file exists [FAILED]
> Hadoop secondarynamenode is running [ OK ]
>
> Thanks,
>
>
Re: namenode is failing
Posted by Harsh J <ha...@cloudera.com>.
If you move the NN dir, you also have to move its older location's
contents there before starting the NN.
On Thu, Feb 28, 2013 at 1:28 PM, Mohit Vadhera
<pr...@gmail.com> wrote:
> Hi Guys,
>
> Namenode switches into safemode when it has low disk space on the root fs /
> i have to manually run a command to leave it
> I have space on other partition. Can I change the path for cache files on
> other partition ? I have below properties . Can it resolve the issue ? If i
> change the path to other directories and restart services I get the below
> error while starting the service namenode. I didn't find anything in logs so
> far. Can you please suggest something ?
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> </property>
> <property>
> <name>dfs.namenode.name.dir</name>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
> </property>
> <property>
> <name>dfs.namenode.checkpoint.dir</name>
>
> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/namesecondary</value>
> </property>
> <property>
>
>
> Service namenode is failing
>
> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
> Hadoop datanode is running [ OK ]
> Hadoop namenode is dead and pid file exists [FAILED]
> Hadoop secondarynamenode is running [ OK ]
>
> Thanks,
>
--
Harsh J