You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Harsh J <ha...@cloudera.com> on 2013/02/28 09:29:26 UTC
Re: NameNode low on available disk space
Hi,
The exact error is displayed on your log and should be somewhat self
explanatory:
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
state: storage directory does not exist or is not accessible.
Please check this one's availability, permissions (the NN user should
be able to access it).
On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
<pr...@gmail.com> wrote:
> Please find below logs for shutting down the namenode service. Can anybody
> check this
>
> 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util: Path
> /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util: Path
> /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 02:07:51,754 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
> of redundant storage directories!
> 2013-02-28 02:07:51,754 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
> 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-02-28 02:07:51,890 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
> 2013-02-28 02:07:51,909 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
> = 512
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
> = 1
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> hdfs (auth:SIMPLE)
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> hadmin
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> 2013-02-28 02:07:51,925 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> 2013-02-28 02:07:52,462 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring
> more than 10 times
> 2013-02-28 02:07:52,466 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> 2013-02-28 02:07:52,467 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
> 2013-02-28 02:07:52,467 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 0
> 2013-02-28 02:07:52,469 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
> 2013-02-28 02:07:52,471 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
> system...
> 2013-02-28 02:07:52,472 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 02:07:52,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 02:07:52,473 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory
> /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
> directory does not exist or is not accessible.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2013-02-28 02:08:48,913 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/192.168.1.3
>
>
> On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
> <pr...@gmail.com> wrote:
>>
>> Hi Guys,
>>
>> I have space on other partition. Can I change the path for cache files on
>> other partition ? I have below properties . Can it resolve the issue ? If i
>> change the path to other directories and restart services I get the below
>> error while starting the service namenode. I didn't find anything in logs so
>> far. Can you please suggest something ?
>>
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>> </property>
>> <property>
>> <name>dfs.namenode.name.dir</name>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>> </property>
>> <property>
>> <name>dfs.namenode.checkpoint.dir</name>
>>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/namesecondary</value>
>> </property>
>> <property>
>>
>>
>> Service namenode is failing
>>
>> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
>> Hadoop datanode is running [ OK ]
>> Hadoop namenode is dead and pid file exists [FAILED]
>> Hadoop secondarynamenode is running [ OK ]
>>
>> Thanks,
>>
>>
>>
>> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>> <pr...@gmail.com> wrote:
>>>
>>>
>>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com> wrote:
>>>>
>>>> http://NNHOST:50070/conf
>>>
>>>
>>>
>>> Harsh, I changed the value as said & restarted service NN. For verifying
>>> i checked the http link that you gave and i saw the property their but on
>>> http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
>>> blocks. Please check the logs or run fsck in order to identify the missing
>>> blocks.) when i clicked on this link i can see file names . Do I need to
>>> reboot the machine to run fsck on root fs/ or is there hadoop command fsck
>>> that i can run on the running hadoop ?
>>>
>>> Thanks,
>>>
>>
>
--
Harsh J
Re: NameNode low on available disk space
Posted by Harsh J <ha...@cloudera.com>.
Actually, I did: http://search-hadoop.com/m/qvn7EhQQHm1. Perhaps you aren't
subscribed to the lists and wouldn't have received anything sent back.
On Fri, Mar 1, 2013 at 1:21 PM, Mohit Vadhera
<pr...@gmail.com>wrote:
> thanks Harsh but you didn't answer on this before, I will try to move old
> directory "name" to new location and restart services . Hope it will not
> loose any data .
>
> old Location
>
> $ sudo ls -l /var/lib/hadoop-hdfs/cache/hdfs/dfs/
> total 12
> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 name
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 namesecondary
>
>
> New location
>
> $ sudo ls -l /mnt/san1/hdfs/hdfs/dfs/
> total 8
> drwx------. 3 hdfs hdfs 4096 Feb 28 11:28 data
> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 11:28 namesecondary
>
>
> Thanks,
>
>
> On Fri, Mar 1, 2013 at 12:14 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> I believe I already responded to another one of your multiple threads
>> on this problem, that you have to move contents of the old dir to the
>> new dir and not format. Thats how one changes the dfs.name.dir
>> properly.
>>
>> On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
>> <pr...@gmail.com> wrote:
>> > Hi,
>> >
>> > May i know the answer plz ?
>> >
>> > Thanks,
>> >
>> >
>> > On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
>> > <pr...@gmail.com> wrote:
>> >>
>> >> Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>> >> and set permission . when i restart hadoop services. It removes and I
>> find
>> >> below logs.
>> >>
>> >> Do I need to format the NN?
>> >> Below is the command to format the NN ?
>> >> Any kind of loss while formatting ?
>> >> Is there any way to avoid formatting and change the cache path ?
>> >>
>> >> 2013-02-28 05:57:50,902 INFO
>> org.apache.hadoop.hdfs.server.common.Storage:
>> >> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by
>> nodename
>> >> 81133@OPERA-MAST1.ny.os.local
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics
>> >> system...
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >> stopped.
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >> shutdown complete.
>> >> 2013-02-28 05:57:50,905 FATAL
>> >> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
>> join
>> >> java.io.IOException: NameNode is not formatted.
>> >>
>> >> Command to format the NN.
>> >>
>> >> sudo -u hdfs hdfs namenode -format
>> >>
>> >> Thanks,
>> >>
>> >>
>> >> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
>> >> <pr...@gmail.com> wrote:
>> >>>
>> >>> After creating the directory and setting permission I tried to restart
>> >>> the services and i get error
>> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>> >>> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are
>> not
>> >>> being started.
>> >>>
>> >>> Need to check few logs from below logs.
>> >>> ===================================
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>> storage
>> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>> to lack
>> >>> of redundant sto
>> >>> rage directories!
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> namespace
>> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> >>> dataloss due to lack of re
>> >>> dundant storage directories!
>> >>>
>> >>>
>> >>> ************************************************************/
>> >>> 2013-02-28 05:06:23,385 WARN
>> >>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate
>> configuration:
>> >>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
>> >>> 2013-02-28 05:06:23,556 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period
>> >>> at 10 second(s).
>> >>> 2013-02-28 05:06:23,556 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >>> started
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>> storage
>> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>> to lack
>> >>> of redundant sto
>> >>> rage directories!
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> namespace
>> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> >>> dataloss due to lack of re
>> >>> dundant storage directories!
>> >>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
>> >>> Refreshing hosts (include/exclude) list
>> >>> 2013-02-28 05:06:25,623 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> >>> dfs.block.invalidate.limit=1000
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> dfs.block.access.token.enable=false
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> defaultReplication = 1
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplication
>> >>> = 512
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> minReplication
>> >>> = 1
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> maxReplicationStreams = 2
>> >>> 2013-02-28 05:06:26,016 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> shouldCheckForEnoughRacks = false
>> >>> 2013-02-28 05:06:26,016 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> replicationRecheckInterval = 3000
>> >>> 2013-02-28 05:06:26,016 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> encryptDataTransfer = false
>> >>> 2013-02-28 05:06:26,022 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>> =
>> >>> hdfs (auth:SIMPLE)
>> >>> 2013-02-28 05:06:26,022 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>> =
>> >>> hadmin
>> >>> 2013-02-28 05:06:26,022 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> isPermissionEnabled =
>> >>> true
>> >>> 2013-02-28 05:06:26,023 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>> >>> 2013-02-28 05:06:26,026 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>> true
>> >>> 2013-02-28 05:06:26,359 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring
>> >>> more than 10 times
>> >>> 2013-02-28 05:06:26,361 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> >>> 2013-02-28 05:06:26,361 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>> dfs.namenode.safemode.min.datanodes = 0
>> >>> 2013-02-28 05:06:26,361 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>> dfs.namenode.safemode.extension = 0
>> >>> 2013-02-28 05:06:26,378 INFO
>> >>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>> >>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> >>> 7275@OPERA-MAST1.ny.os.local
>> >>> 2013-02-28 05:06:26,381 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics
>> >>> system...
>> >>> 2013-02-28 05:06:26,381 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >>> stopped.
>> >>> 2013-02-28 05:06:26,381 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >>> shutdown complete.
>> >>> 2013-02-28 05:06:26,382 FATAL
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>> namenode join
>> >>> java.io.IOException: NameNode is not formatted.
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> >>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> >>> with status 1
>> >>> 2013-02-28 05:06:26,394 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> >>> /************************************************************
>> >>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
>> >>> <pr...@gmail.com> wrote:
>> >>>>
>> >>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being
>> created .
>> >>>> If I do compare with the older path the permissions are same on the
>> parent
>> >>>> directories.
>> >>>> Do I need to create this this directory manually and set the
>> permission
>> >>>> ?
>> >>>>
>> >>>> Older Path
>> >>>>
>> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>> >>>> total 4
>> >>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>> >>>>
>> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>> >>>> total 12
>> >>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>> >>>>
>> >>>>
>> >>>> New Path
>> >>>>
>> >>>> # ll /mnt/san1/hdfs/cache/hdfs/
>> >>>> total 4
>> >>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>> >>>>
>> >>>>
>> >>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>> >>>> total 4
>> >>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>> >>>>
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> The exact error is displayed on your log and should be somewhat self
>> >>>>> explanatory:
>> >>>>>
>> >>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> >>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>> >>>>> state: storage directory does not exist or is not accessible.
>> >>>>>
>> >>>>> Please check this one's availability, permissions (the NN user
>> should
>> >>>>> be able to access it).
>> >>>>>
>> >>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>> >>>>> <pr...@gmail.com> wrote:
>> >>>>> > Please find below logs for shutting down the namenode service. Can
>> >>>>> > anybody
>> >>>>> > check this
>> >>>>> >
>> >>>>> > 2013-02-28 02:07:51,752 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> >>>>> > configuration files. Please update hdfs configuration.
>> >>>>> > 2013-02-28 02:07:51,754 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> >>>>> > configuration files. Please update hdfs configuration.
>> >>>>> > 2013-02-28 02:07:51,754 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> image
>> >>>>> > storage
>> >>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss
>> due
>> >>>>> > to lack
>> >>>>> > of redundant storage directories!
>> >>>>> > 2013-02-28 02:07:51,754 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> >>>>> > namespace
>> >>>>> > edits storage directory (dfs.namenode.edits.dir) configured.
>> Beware
>> >>>>> > of
>> >>>>> > dataloss due to lack of redundant storage directories!
>> >>>>> > 2013-02-28 02:07:51,884 INFO
>> org.apache.hadoop.util.HostsFileReader:
>> >>>>> > Refreshing hosts (include/exclude) list
>> >>>>> > 2013-02-28 02:07:51,890 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> >>>>> > dfs.block.invalidate.limit=1000
>> >>>>> > 2013-02-28 02:07:51,909 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > dfs.block.access.token.enable=false
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > defaultReplication = 1
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > maxReplication
>> >>>>> > = 512
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > minReplication
>> >>>>> > = 1
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > maxReplicationStreams = 2
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > shouldCheckForEnoughRacks = false
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > replicationRecheckInterval = 3000
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > encryptDataTransfer = false
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>> >>>>> > =
>> >>>>> > hdfs (auth:SIMPLE)
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>> >>>>> > =
>> >>>>> > hadmin
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > isPermissionEnabled =
>> >>>>> > true
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
>> >>>>> > false
>> >>>>> > 2013-02-28 02:07:51,925 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
>> Enabled:
>> >>>>> > true
>> >>>>> > 2013-02-28 02:07:52,462 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file
>> names
>> >>>>> > occuring
>> >>>>> > more than 10 times
>> >>>>> > 2013-02-28 02:07:52,466 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> >>>>> > 2013-02-28 02:07:52,467 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > dfs.namenode.safemode.min.datanodes = 0
>> >>>>> > 2013-02-28 02:07:52,467 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > dfs.namenode.safemode.extension = 0
>> >>>>> > 2013-02-28 02:07:52,469 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.common.Storage:
>> >>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not
>> exist.
>> >>>>> > 2013-02-28 02:07:52,471 INFO
>> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping
>> NameNode
>> >>>>> > metrics
>> >>>>> > system...
>> >>>>> > 2013-02-28 02:07:52,472 INFO
>> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
>> metrics
>> >>>>> > system
>> >>>>> > stopped.
>> >>>>> > 2013-02-28 02:07:52,473 INFO
>> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
>> metrics
>> >>>>> > system
>> >>>>> > shutdown complete.
>> >>>>> > 2013-02-28 02:07:52,473 FATAL
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>> >>>>> > namenode join
>> >>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> >>>>> > Directory
>> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
>> >>>>> > storage
>> >>>>> > directory does not exist or is not accessible.
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> >>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil:
>> Exiting
>> >>>>> > with
>> >>>>> > status 1
>> >>>>> > 2013-02-28 02:08:48,913 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> >>>>> > /************************************************************
>> >>>>> > SHUTDOWN_MSG: Shutting down NameNode at
>> >>>>> > OPERA-MAST1.ny.os.local/192.168.1.3
>> >>>>> >
>> >>>>> >
>> >>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>> >>>>> > <pr...@gmail.com> wrote:
>> >>>>> >>
>> >>>>> >> Hi Guys,
>> >>>>> >>
>> >>>>> >> I have space on other partition. Can I change the path for cache
>> >>>>> >> files on
>> >>>>> >> other partition ? I have below properties . Can it resolve the
>> issue
>> >>>>> >> ? If i
>> >>>>> >> change the path to other directories and restart services I get
>> the
>> >>>>> >> below
>> >>>>> >> error while starting the service namenode. I didn't find
>> anything in
>> >>>>> >> logs so
>> >>>>> >> far. Can you please suggest something ?
>> >>>>> >>
>> >>>>> >> <property>
>> >>>>> >> <name>hadoop.tmp.dir</name>
>> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>> >>>>> >> </property>
>> >>>>> >> <property>
>> >>>>> >> <name>dfs.namenode.name.dir</name>
>> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/name</value>
>> >>>>> >> </property>
>> >>>>> >> <property>
>> >>>>> >> <name>dfs.namenode.checkpoint.dir</name>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/namesecondary</value>
>> >>>>> >> </property>
>> >>>>> >> <property>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> Service namenode is failing
>> >>>>> >>
>> >>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
>> >>>>> >> status; done
>> >>>>> >> Hadoop datanode is running [ OK
>> ]
>> >>>>> >> Hadoop namenode is dead and pid file exists
>> [FAILED]
>> >>>>> >> Hadoop secondarynamenode is running [ OK
>> ]
>> >>>>> >>
>> >>>>> >> Thanks,
>> >>>>> >>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>> >>>>> >> <pr...@gmail.com> wrote:
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>> >>>>> >>> wrote:
>> >>>>> >>>>
>> >>>>> >>>> http://NNHOST:50070/conf
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>> Harsh, I changed the value as said & restarted service NN. For
>> >>>>> >>> verifying
>> >>>>> >>> i checked the http link that you gave and i saw the property
>> their
>> >>>>> >>> but on
>> >>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>> >>>>> >>> missing
>> >>>>> >>> blocks. Please check the logs or run fsck in order to identify
>> the
>> >>>>> >>> missing
>> >>>>> >>> blocks.) when i clicked on this link i can see file names .
>> Do I
>> >>>>> >>> need to
>> >>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>> >>>>> >>> command fsck
>> >>>>> >>> that i can run on the running hadoop ?
>> >>>>> >>>
>> >>>>> >>> Thanks,
>> >>>>> >>>
>> >>>>> >>
>> >>>>> >
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Harsh J
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Harsh J
Re: NameNode low on available disk space
Posted by Harsh J <ha...@cloudera.com>.
Actually, I did: http://search-hadoop.com/m/qvn7EhQQHm1. Perhaps you aren't
subscribed to the lists and wouldn't have received anything sent back.
On Fri, Mar 1, 2013 at 1:21 PM, Mohit Vadhera
<pr...@gmail.com>wrote:
> thanks Harsh but you didn't answer on this before, I will try to move old
> directory "name" to new location and restart services . Hope it will not
> loose any data .
>
> old Location
>
> $ sudo ls -l /var/lib/hadoop-hdfs/cache/hdfs/dfs/
> total 12
> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 name
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 namesecondary
>
>
> New location
>
> $ sudo ls -l /mnt/san1/hdfs/hdfs/dfs/
> total 8
> drwx------. 3 hdfs hdfs 4096 Feb 28 11:28 data
> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 11:28 namesecondary
>
>
> Thanks,
>
>
> On Fri, Mar 1, 2013 at 12:14 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> I believe I already responded to another one of your multiple threads
>> on this problem, that you have to move contents of the old dir to the
>> new dir and not format. Thats how one changes the dfs.name.dir
>> properly.
>>
>> On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
>> <pr...@gmail.com> wrote:
>> > Hi,
>> >
>> > May i know the answer plz ?
>> >
>> > Thanks,
>> >
>> >
>> > On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
>> > <pr...@gmail.com> wrote:
>> >>
>> >> Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>> >> and set permission . when i restart hadoop services. It removes and I
>> find
>> >> below logs.
>> >>
>> >> Do I need to format the NN?
>> >> Below is the command to format the NN ?
>> >> Any kind of loss while formatting ?
>> >> Is there any way to avoid formatting and change the cache path ?
>> >>
>> >> 2013-02-28 05:57:50,902 INFO
>> org.apache.hadoop.hdfs.server.common.Storage:
>> >> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by
>> nodename
>> >> 81133@OPERA-MAST1.ny.os.local
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics
>> >> system...
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >> stopped.
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >> shutdown complete.
>> >> 2013-02-28 05:57:50,905 FATAL
>> >> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
>> join
>> >> java.io.IOException: NameNode is not formatted.
>> >>
>> >> Command to format the NN.
>> >>
>> >> sudo -u hdfs hdfs namenode -format
>> >>
>> >> Thanks,
>> >>
>> >>
>> >> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
>> >> <pr...@gmail.com> wrote:
>> >>>
>> >>> After creating the directory and setting permission I tried to restart
>> >>> the services and i get error
>> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>> >>> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are
>> not
>> >>> being started.
>> >>>
>> >>> Need to check few logs from below logs.
>> >>> ===================================
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>> storage
>> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>> to lack
>> >>> of redundant sto
>> >>> rage directories!
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> namespace
>> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> >>> dataloss due to lack of re
>> >>> dundant storage directories!
>> >>>
>> >>>
>> >>> ************************************************************/
>> >>> 2013-02-28 05:06:23,385 WARN
>> >>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate
>> configuration:
>> >>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
>> >>> 2013-02-28 05:06:23,556 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period
>> >>> at 10 second(s).
>> >>> 2013-02-28 05:06:23,556 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >>> started
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>> storage
>> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>> to lack
>> >>> of redundant sto
>> >>> rage directories!
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> namespace
>> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> >>> dataloss due to lack of re
>> >>> dundant storage directories!
>> >>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
>> >>> Refreshing hosts (include/exclude) list
>> >>> 2013-02-28 05:06:25,623 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> >>> dfs.block.invalidate.limit=1000
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> dfs.block.access.token.enable=false
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> defaultReplication = 1
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplication
>> >>> = 512
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> minReplication
>> >>> = 1
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> maxReplicationStreams = 2
>> >>> 2013-02-28 05:06:26,016 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> shouldCheckForEnoughRacks = false
>> >>> 2013-02-28 05:06:26,016 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> replicationRecheckInterval = 3000
>> >>> 2013-02-28 05:06:26,016 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> encryptDataTransfer = false
>> >>> 2013-02-28 05:06:26,022 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>> =
>> >>> hdfs (auth:SIMPLE)
>> >>> 2013-02-28 05:06:26,022 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>> =
>> >>> hadmin
>> >>> 2013-02-28 05:06:26,022 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> isPermissionEnabled =
>> >>> true
>> >>> 2013-02-28 05:06:26,023 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>> >>> 2013-02-28 05:06:26,026 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>> true
>> >>> 2013-02-28 05:06:26,359 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring
>> >>> more than 10 times
>> >>> 2013-02-28 05:06:26,361 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> >>> 2013-02-28 05:06:26,361 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>> dfs.namenode.safemode.min.datanodes = 0
>> >>> 2013-02-28 05:06:26,361 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>> dfs.namenode.safemode.extension = 0
>> >>> 2013-02-28 05:06:26,378 INFO
>> >>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>> >>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> >>> 7275@OPERA-MAST1.ny.os.local
>> >>> 2013-02-28 05:06:26,381 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics
>> >>> system...
>> >>> 2013-02-28 05:06:26,381 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >>> stopped.
>> >>> 2013-02-28 05:06:26,381 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >>> shutdown complete.
>> >>> 2013-02-28 05:06:26,382 FATAL
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>> namenode join
>> >>> java.io.IOException: NameNode is not formatted.
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> >>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> >>> with status 1
>> >>> 2013-02-28 05:06:26,394 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> >>> /************************************************************
>> >>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
>> >>> <pr...@gmail.com> wrote:
>> >>>>
>> >>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being
>> created .
>> >>>> If I do compare with the older path the permissions are same on the
>> parent
>> >>>> directories.
>> >>>> Do I need to create this this directory manually and set the
>> permission
>> >>>> ?
>> >>>>
>> >>>> Older Path
>> >>>>
>> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>> >>>> total 4
>> >>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>> >>>>
>> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>> >>>> total 12
>> >>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>> >>>>
>> >>>>
>> >>>> New Path
>> >>>>
>> >>>> # ll /mnt/san1/hdfs/cache/hdfs/
>> >>>> total 4
>> >>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>> >>>>
>> >>>>
>> >>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>> >>>> total 4
>> >>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>> >>>>
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> The exact error is displayed on your log and should be somewhat self
>> >>>>> explanatory:
>> >>>>>
>> >>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> >>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>> >>>>> state: storage directory does not exist or is not accessible.
>> >>>>>
>> >>>>> Please check this one's availability, permissions (the NN user
>> should
>> >>>>> be able to access it).
>> >>>>>
>> >>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>> >>>>> <pr...@gmail.com> wrote:
>> >>>>> > Please find below logs for shutting down the namenode service. Can
>> >>>>> > anybody
>> >>>>> > check this
>> >>>>> >
>> >>>>> > 2013-02-28 02:07:51,752 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> >>>>> > configuration files. Please update hdfs configuration.
>> >>>>> > 2013-02-28 02:07:51,754 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> >>>>> > configuration files. Please update hdfs configuration.
>> >>>>> > 2013-02-28 02:07:51,754 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> image
>> >>>>> > storage
>> >>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss
>> due
>> >>>>> > to lack
>> >>>>> > of redundant storage directories!
>> >>>>> > 2013-02-28 02:07:51,754 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> >>>>> > namespace
>> >>>>> > edits storage directory (dfs.namenode.edits.dir) configured.
>> Beware
>> >>>>> > of
>> >>>>> > dataloss due to lack of redundant storage directories!
>> >>>>> > 2013-02-28 02:07:51,884 INFO
>> org.apache.hadoop.util.HostsFileReader:
>> >>>>> > Refreshing hosts (include/exclude) list
>> >>>>> > 2013-02-28 02:07:51,890 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> >>>>> > dfs.block.invalidate.limit=1000
>> >>>>> > 2013-02-28 02:07:51,909 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > dfs.block.access.token.enable=false
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > defaultReplication = 1
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > maxReplication
>> >>>>> > = 512
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > minReplication
>> >>>>> > = 1
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > maxReplicationStreams = 2
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > shouldCheckForEnoughRacks = false
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > replicationRecheckInterval = 3000
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > encryptDataTransfer = false
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>> >>>>> > =
>> >>>>> > hdfs (auth:SIMPLE)
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>> >>>>> > =
>> >>>>> > hadmin
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > isPermissionEnabled =
>> >>>>> > true
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
>> >>>>> > false
>> >>>>> > 2013-02-28 02:07:51,925 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
>> Enabled:
>> >>>>> > true
>> >>>>> > 2013-02-28 02:07:52,462 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file
>> names
>> >>>>> > occuring
>> >>>>> > more than 10 times
>> >>>>> > 2013-02-28 02:07:52,466 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> >>>>> > 2013-02-28 02:07:52,467 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > dfs.namenode.safemode.min.datanodes = 0
>> >>>>> > 2013-02-28 02:07:52,467 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > dfs.namenode.safemode.extension = 0
>> >>>>> > 2013-02-28 02:07:52,469 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.common.Storage:
>> >>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not
>> exist.
>> >>>>> > 2013-02-28 02:07:52,471 INFO
>> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping
>> NameNode
>> >>>>> > metrics
>> >>>>> > system...
>> >>>>> > 2013-02-28 02:07:52,472 INFO
>> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
>> metrics
>> >>>>> > system
>> >>>>> > stopped.
>> >>>>> > 2013-02-28 02:07:52,473 INFO
>> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
>> metrics
>> >>>>> > system
>> >>>>> > shutdown complete.
>> >>>>> > 2013-02-28 02:07:52,473 FATAL
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>> >>>>> > namenode join
>> >>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> >>>>> > Directory
>> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
>> >>>>> > storage
>> >>>>> > directory does not exist or is not accessible.
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> >>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil:
>> Exiting
>> >>>>> > with
>> >>>>> > status 1
>> >>>>> > 2013-02-28 02:08:48,913 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> >>>>> > /************************************************************
>> >>>>> > SHUTDOWN_MSG: Shutting down NameNode at
>> >>>>> > OPERA-MAST1.ny.os.local/192.168.1.3
>> >>>>> >
>> >>>>> >
>> >>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>> >>>>> > <pr...@gmail.com> wrote:
>> >>>>> >>
>> >>>>> >> Hi Guys,
>> >>>>> >>
>> >>>>> >> I have space on other partition. Can I change the path for cache
>> >>>>> >> files on
>> >>>>> >> other partition ? I have below properties . Can it resolve the
>> issue
>> >>>>> >> ? If i
>> >>>>> >> change the path to other directories and restart services I get
>> the
>> >>>>> >> below
>> >>>>> >> error while starting the service namenode. I didn't find
>> anything in
>> >>>>> >> logs so
>> >>>>> >> far. Can you please suggest something ?
>> >>>>> >>
>> >>>>> >> <property>
>> >>>>> >> <name>hadoop.tmp.dir</name>
>> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>> >>>>> >> </property>
>> >>>>> >> <property>
>> >>>>> >> <name>dfs.namenode.name.dir</name>
>> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/name</value>
>> >>>>> >> </property>
>> >>>>> >> <property>
>> >>>>> >> <name>dfs.namenode.checkpoint.dir</name>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/namesecondary</value>
>> >>>>> >> </property>
>> >>>>> >> <property>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> Service namenode is failing
>> >>>>> >>
>> >>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
>> >>>>> >> status; done
>> >>>>> >> Hadoop datanode is running [ OK
>> ]
>> >>>>> >> Hadoop namenode is dead and pid file exists
>> [FAILED]
>> >>>>> >> Hadoop secondarynamenode is running [ OK
>> ]
>> >>>>> >>
>> >>>>> >> Thanks,
>> >>>>> >>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>> >>>>> >> <pr...@gmail.com> wrote:
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>> >>>>> >>> wrote:
>> >>>>> >>>>
>> >>>>> >>>> http://NNHOST:50070/conf
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>> Harsh, I changed the value as said & restarted service NN. For
>> >>>>> >>> verifying
>> >>>>> >>> i checked the http link that you gave and i saw the property
>> their
>> >>>>> >>> but on
>> >>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>> >>>>> >>> missing
>> >>>>> >>> blocks. Please check the logs or run fsck in order to identify
>> the
>> >>>>> >>> missing
>> >>>>> >>> blocks.) when i clicked on this link i can see file names .
>> Do I
>> >>>>> >>> need to
>> >>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>> >>>>> >>> command fsck
>> >>>>> >>> that i can run on the running hadoop ?
>> >>>>> >>>
>> >>>>> >>> Thanks,
>> >>>>> >>>
>> >>>>> >>
>> >>>>> >
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Harsh J
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Harsh J
Re: NameNode low on available disk space
Posted by Harsh J <ha...@cloudera.com>.
Actually, I did: http://search-hadoop.com/m/qvn7EhQQHm1. Perhaps you aren't
subscribed to the lists and wouldn't have received anything sent back.
On Fri, Mar 1, 2013 at 1:21 PM, Mohit Vadhera
<pr...@gmail.com>wrote:
> thanks Harsh but you didn't answer on this before, I will try to move old
> directory "name" to new location and restart services . Hope it will not
> loose any data .
>
> old Location
>
> $ sudo ls -l /var/lib/hadoop-hdfs/cache/hdfs/dfs/
> total 12
> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 name
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 namesecondary
>
>
> New location
>
> $ sudo ls -l /mnt/san1/hdfs/hdfs/dfs/
> total 8
> drwx------. 3 hdfs hdfs 4096 Feb 28 11:28 data
> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 11:28 namesecondary
>
>
> Thanks,
>
>
> On Fri, Mar 1, 2013 at 12:14 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> I believe I already responded to another one of your multiple threads
>> on this problem, that you have to move contents of the old dir to the
>> new dir and not format. Thats how one changes the dfs.name.dir
>> properly.
>>
>> On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
>> <pr...@gmail.com> wrote:
>> > Hi,
>> >
>> > May i know the answer plz ?
>> >
>> > Thanks,
>> >
>> >
>> > On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
>> > <pr...@gmail.com> wrote:
>> >>
>> >> Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>> >> and set permission . when i restart hadoop services. It removes and I
>> find
>> >> below logs.
>> >>
>> >> Do I need to format the NN?
>> >> Below is the command to format the NN ?
>> >> Any kind of loss while formatting ?
>> >> Is there any way to avoid formatting and change the cache path ?
>> >>
>> >> 2013-02-28 05:57:50,902 INFO
>> org.apache.hadoop.hdfs.server.common.Storage:
>> >> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by
>> nodename
>> >> 81133@OPERA-MAST1.ny.os.local
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics
>> >> system...
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >> stopped.
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >> shutdown complete.
>> >> 2013-02-28 05:57:50,905 FATAL
>> >> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
>> join
>> >> java.io.IOException: NameNode is not formatted.
>> >>
>> >> Command to format the NN.
>> >>
>> >> sudo -u hdfs hdfs namenode -format
>> >>
>> >> Thanks,
>> >>
>> >>
>> >> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
>> >> <pr...@gmail.com> wrote:
>> >>>
>> >>> After creating the directory and setting permission I tried to restart
>> >>> the services and i get error
>> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>> >>> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are
>> not
>> >>> being started.
>> >>>
>> >>> Need to check few logs from below logs.
>> >>> ===================================
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>> storage
>> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>> to lack
>> >>> of redundant sto
>> >>> rage directories!
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> namespace
>> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> >>> dataloss due to lack of re
>> >>> dundant storage directories!
>> >>>
>> >>>
>> >>> ************************************************************/
>> >>> 2013-02-28 05:06:23,385 WARN
>> >>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate
>> configuration:
>> >>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
>> >>> 2013-02-28 05:06:23,556 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period
>> >>> at 10 second(s).
>> >>> 2013-02-28 05:06:23,556 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >>> started
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>> storage
>> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>> to lack
>> >>> of redundant sto
>> >>> rage directories!
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> namespace
>> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> >>> dataloss due to lack of re
>> >>> dundant storage directories!
>> >>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
>> >>> Refreshing hosts (include/exclude) list
>> >>> 2013-02-28 05:06:25,623 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> >>> dfs.block.invalidate.limit=1000
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> dfs.block.access.token.enable=false
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> defaultReplication = 1
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplication
>> >>> = 512
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> minReplication
>> >>> = 1
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> maxReplicationStreams = 2
>> >>> 2013-02-28 05:06:26,016 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> shouldCheckForEnoughRacks = false
>> >>> 2013-02-28 05:06:26,016 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> replicationRecheckInterval = 3000
>> >>> 2013-02-28 05:06:26,016 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> encryptDataTransfer = false
>> >>> 2013-02-28 05:06:26,022 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>> =
>> >>> hdfs (auth:SIMPLE)
>> >>> 2013-02-28 05:06:26,022 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>> =
>> >>> hadmin
>> >>> 2013-02-28 05:06:26,022 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> isPermissionEnabled =
>> >>> true
>> >>> 2013-02-28 05:06:26,023 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>> >>> 2013-02-28 05:06:26,026 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>> true
>> >>> 2013-02-28 05:06:26,359 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring
>> >>> more than 10 times
>> >>> 2013-02-28 05:06:26,361 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> >>> 2013-02-28 05:06:26,361 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>> dfs.namenode.safemode.min.datanodes = 0
>> >>> 2013-02-28 05:06:26,361 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>> dfs.namenode.safemode.extension = 0
>> >>> 2013-02-28 05:06:26,378 INFO
>> >>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>> >>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> >>> 7275@OPERA-MAST1.ny.os.local
>> >>> 2013-02-28 05:06:26,381 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics
>> >>> system...
>> >>> 2013-02-28 05:06:26,381 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >>> stopped.
>> >>> 2013-02-28 05:06:26,381 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >>> shutdown complete.
>> >>> 2013-02-28 05:06:26,382 FATAL
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>> namenode join
>> >>> java.io.IOException: NameNode is not formatted.
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> >>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> >>> with status 1
>> >>> 2013-02-28 05:06:26,394 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> >>> /************************************************************
>> >>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
>> >>> <pr...@gmail.com> wrote:
>> >>>>
>> >>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being
>> created .
>> >>>> If I do compare with the older path the permissions are same on the
>> parent
>> >>>> directories.
>> >>>> Do I need to create this this directory manually and set the
>> permission
>> >>>> ?
>> >>>>
>> >>>> Older Path
>> >>>>
>> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>> >>>> total 4
>> >>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>> >>>>
>> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>> >>>> total 12
>> >>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>> >>>>
>> >>>>
>> >>>> New Path
>> >>>>
>> >>>> # ll /mnt/san1/hdfs/cache/hdfs/
>> >>>> total 4
>> >>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>> >>>>
>> >>>>
>> >>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>> >>>> total 4
>> >>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>> >>>>
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> The exact error is displayed on your log and should be somewhat self
>> >>>>> explanatory:
>> >>>>>
>> >>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> >>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>> >>>>> state: storage directory does not exist or is not accessible.
>> >>>>>
>> >>>>> Please check this one's availability, permissions (the NN user
>> should
>> >>>>> be able to access it).
>> >>>>>
>> >>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>> >>>>> <pr...@gmail.com> wrote:
>> >>>>> > Please find below logs for shutting down the namenode service. Can
>> >>>>> > anybody
>> >>>>> > check this
>> >>>>> >
>> >>>>> > 2013-02-28 02:07:51,752 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> >>>>> > configuration files. Please update hdfs configuration.
>> >>>>> > 2013-02-28 02:07:51,754 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> >>>>> > configuration files. Please update hdfs configuration.
>> >>>>> > 2013-02-28 02:07:51,754 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> image
>> >>>>> > storage
>> >>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss
>> due
>> >>>>> > to lack
>> >>>>> > of redundant storage directories!
>> >>>>> > 2013-02-28 02:07:51,754 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> >>>>> > namespace
>> >>>>> > edits storage directory (dfs.namenode.edits.dir) configured.
>> Beware
>> >>>>> > of
>> >>>>> > dataloss due to lack of redundant storage directories!
>> >>>>> > 2013-02-28 02:07:51,884 INFO
>> org.apache.hadoop.util.HostsFileReader:
>> >>>>> > Refreshing hosts (include/exclude) list
>> >>>>> > 2013-02-28 02:07:51,890 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> >>>>> > dfs.block.invalidate.limit=1000
>> >>>>> > 2013-02-28 02:07:51,909 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > dfs.block.access.token.enable=false
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > defaultReplication = 1
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > maxReplication
>> >>>>> > = 512
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > minReplication
>> >>>>> > = 1
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > maxReplicationStreams = 2
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > shouldCheckForEnoughRacks = false
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > replicationRecheckInterval = 3000
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > encryptDataTransfer = false
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>> >>>>> > =
>> >>>>> > hdfs (auth:SIMPLE)
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>> >>>>> > =
>> >>>>> > hadmin
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > isPermissionEnabled =
>> >>>>> > true
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
>> >>>>> > false
>> >>>>> > 2013-02-28 02:07:51,925 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
>> Enabled:
>> >>>>> > true
>> >>>>> > 2013-02-28 02:07:52,462 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file
>> names
>> >>>>> > occuring
>> >>>>> > more than 10 times
>> >>>>> > 2013-02-28 02:07:52,466 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> >>>>> > 2013-02-28 02:07:52,467 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > dfs.namenode.safemode.min.datanodes = 0
>> >>>>> > 2013-02-28 02:07:52,467 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > dfs.namenode.safemode.extension = 0
>> >>>>> > 2013-02-28 02:07:52,469 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.common.Storage:
>> >>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not
>> exist.
>> >>>>> > 2013-02-28 02:07:52,471 INFO
>> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping
>> NameNode
>> >>>>> > metrics
>> >>>>> > system...
>> >>>>> > 2013-02-28 02:07:52,472 INFO
>> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
>> metrics
>> >>>>> > system
>> >>>>> > stopped.
>> >>>>> > 2013-02-28 02:07:52,473 INFO
>> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
>> metrics
>> >>>>> > system
>> >>>>> > shutdown complete.
>> >>>>> > 2013-02-28 02:07:52,473 FATAL
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>> >>>>> > namenode join
>> >>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> >>>>> > Directory
>> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
>> >>>>> > storage
>> >>>>> > directory does not exist or is not accessible.
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> >>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil:
>> Exiting
>> >>>>> > with
>> >>>>> > status 1
>> >>>>> > 2013-02-28 02:08:48,913 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> >>>>> > /************************************************************
>> >>>>> > SHUTDOWN_MSG: Shutting down NameNode at
>> >>>>> > OPERA-MAST1.ny.os.local/192.168.1.3
>> >>>>> >
>> >>>>> >
>> >>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>> >>>>> > <pr...@gmail.com> wrote:
>> >>>>> >>
>> >>>>> >> Hi Guys,
>> >>>>> >>
>> >>>>> >> I have space on other partition. Can I change the path for cache
>> >>>>> >> files on
>> >>>>> >> other partition ? I have below properties . Can it resolve the
>> issue
>> >>>>> >> ? If i
>> >>>>> >> change the path to other directories and restart services I get
>> the
>> >>>>> >> below
>> >>>>> >> error while starting the service namenode. I didn't find
>> anything in
>> >>>>> >> logs so
>> >>>>> >> far. Can you please suggest something ?
>> >>>>> >>
>> >>>>> >> <property>
>> >>>>> >> <name>hadoop.tmp.dir</name>
>> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>> >>>>> >> </property>
>> >>>>> >> <property>
>> >>>>> >> <name>dfs.namenode.name.dir</name>
>> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/name</value>
>> >>>>> >> </property>
>> >>>>> >> <property>
>> >>>>> >> <name>dfs.namenode.checkpoint.dir</name>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/namesecondary</value>
>> >>>>> >> </property>
>> >>>>> >> <property>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> Service namenode is failing
>> >>>>> >>
>> >>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
>> >>>>> >> status; done
>> >>>>> >> Hadoop datanode is running [ OK
>> ]
>> >>>>> >> Hadoop namenode is dead and pid file exists
>> [FAILED]
>> >>>>> >> Hadoop secondarynamenode is running [ OK
>> ]
>> >>>>> >>
>> >>>>> >> Thanks,
>> >>>>> >>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>> >>>>> >> <pr...@gmail.com> wrote:
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>> >>>>> >>> wrote:
>> >>>>> >>>>
>> >>>>> >>>> http://NNHOST:50070/conf
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>> Harsh, I changed the value as said & restarted service NN. For
>> >>>>> >>> verifying
>> >>>>> >>> i checked the http link that you gave and i saw the property
>> their
>> >>>>> >>> but on
>> >>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>> >>>>> >>> missing
>> >>>>> >>> blocks. Please check the logs or run fsck in order to identify
>> the
>> >>>>> >>> missing
>> >>>>> >>> blocks.) when i clicked on this link i can see file names .
>> Do I
>> >>>>> >>> need to
>> >>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>> >>>>> >>> command fsck
>> >>>>> >>> that i can run on the running hadoop ?
>> >>>>> >>>
>> >>>>> >>> Thanks,
>> >>>>> >>>
>> >>>>> >>
>> >>>>> >
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Harsh J
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Harsh J
Re: NameNode low on available disk space
Posted by Harsh J <ha...@cloudera.com>.
Actually, I did: http://search-hadoop.com/m/qvn7EhQQHm1. Perhaps you aren't
subscribed to the lists and wouldn't have received anything sent back.
On Fri, Mar 1, 2013 at 1:21 PM, Mohit Vadhera
<pr...@gmail.com>wrote:
> thanks Harsh but you didn't answer on this before, I will try to move old
> directory "name" to new location and restart services . Hope it will not
> loose any data .
>
> old Location
>
> $ sudo ls -l /var/lib/hadoop-hdfs/cache/hdfs/dfs/
> total 12
> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 name
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 namesecondary
>
>
> New location
>
> $ sudo ls -l /mnt/san1/hdfs/hdfs/dfs/
> total 8
> drwx------. 3 hdfs hdfs 4096 Feb 28 11:28 data
> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 11:28 namesecondary
>
>
> Thanks,
>
>
> On Fri, Mar 1, 2013 at 12:14 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> I believe I already responded to another one of your multiple threads
>> on this problem, that you have to move contents of the old dir to the
>> new dir and not format. Thats how one changes the dfs.name.dir
>> properly.
>>
>> On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
>> <pr...@gmail.com> wrote:
>> > Hi,
>> >
>> > May i know the answer plz ?
>> >
>> > Thanks,
>> >
>> >
>> > On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
>> > <pr...@gmail.com> wrote:
>> >>
>> >> Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>> >> and set permission . when i restart hadoop services. It removes and I
>> find
>> >> below logs.
>> >>
>> >> Do I need to format the NN?
>> >> Below is the command to format the NN ?
>> >> Any kind of loss while formatting ?
>> >> Is there any way to avoid formatting and change the cache path ?
>> >>
>> >> 2013-02-28 05:57:50,902 INFO
>> org.apache.hadoop.hdfs.server.common.Storage:
>> >> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by
>> nodename
>> >> 81133@OPERA-MAST1.ny.os.local
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics
>> >> system...
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >> stopped.
>> >> 2013-02-28 05:57:50,904 INFO
>> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >> shutdown complete.
>> >> 2013-02-28 05:57:50,905 FATAL
>> >> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
>> join
>> >> java.io.IOException: NameNode is not formatted.
>> >>
>> >> Command to format the NN.
>> >>
>> >> sudo -u hdfs hdfs namenode -format
>> >>
>> >> Thanks,
>> >>
>> >>
>> >> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
>> >> <pr...@gmail.com> wrote:
>> >>>
>> >>> After creating the directory and setting permission I tried to restart
>> >>> the services and i get error
>> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>> >>> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are
>> not
>> >>> being started.
>> >>>
>> >>> Need to check few logs from below logs.
>> >>> ===================================
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>> storage
>> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>> to lack
>> >>> of redundant sto
>> >>> rage directories!
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> namespace
>> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> >>> dataloss due to lack of re
>> >>> dundant storage directories!
>> >>>
>> >>>
>> >>> ************************************************************/
>> >>> 2013-02-28 05:06:23,385 WARN
>> >>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate
>> configuration:
>> >>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
>> >>> 2013-02-28 05:06:23,556 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period
>> >>> at 10 second(s).
>> >>> 2013-02-28 05:06:23,556 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >>> started
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,905 WARN
>> org.apache.hadoop.hdfs.server.common.Util:
>> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
>> in
>> >>> configuration files. Please update hdfs configuration.
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>> storage
>> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>> to lack
>> >>> of redundant sto
>> >>> rage directories!
>> >>> 2013-02-28 05:06:24,906 WARN
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> namespace
>> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> >>> dataloss due to lack of re
>> >>> dundant storage directories!
>> >>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
>> >>> Refreshing hosts (include/exclude) list
>> >>> 2013-02-28 05:06:25,623 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> >>> dfs.block.invalidate.limit=1000
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> dfs.block.access.token.enable=false
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> defaultReplication = 1
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplication
>> >>> = 512
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> minReplication
>> >>> = 1
>> >>> 2013-02-28 05:06:26,015 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> maxReplicationStreams = 2
>> >>> 2013-02-28 05:06:26,016 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> shouldCheckForEnoughRacks = false
>> >>> 2013-02-28 05:06:26,016 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> replicationRecheckInterval = 3000
>> >>> 2013-02-28 05:06:26,016 INFO
>> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>> encryptDataTransfer = false
>> >>> 2013-02-28 05:06:26,022 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>> =
>> >>> hdfs (auth:SIMPLE)
>> >>> 2013-02-28 05:06:26,022 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>> =
>> >>> hadmin
>> >>> 2013-02-28 05:06:26,022 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> isPermissionEnabled =
>> >>> true
>> >>> 2013-02-28 05:06:26,023 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>> >>> 2013-02-28 05:06:26,026 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>> true
>> >>> 2013-02-28 05:06:26,359 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring
>> >>> more than 10 times
>> >>> 2013-02-28 05:06:26,361 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> >>> 2013-02-28 05:06:26,361 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>> dfs.namenode.safemode.min.datanodes = 0
>> >>> 2013-02-28 05:06:26,361 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>> dfs.namenode.safemode.extension = 0
>> >>> 2013-02-28 05:06:26,378 INFO
>> >>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>> >>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> >>> 7275@OPERA-MAST1.ny.os.local
>> >>> 2013-02-28 05:06:26,381 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics
>> >>> system...
>> >>> 2013-02-28 05:06:26,381 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >>> stopped.
>> >>> 2013-02-28 05:06:26,381 INFO
>> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> >>> shutdown complete.
>> >>> 2013-02-28 05:06:26,382 FATAL
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>> namenode join
>> >>> java.io.IOException: NameNode is not formatted.
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> >>> at
>> >>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> >>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> >>> with status 1
>> >>> 2013-02-28 05:06:26,394 INFO
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> >>> /************************************************************
>> >>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
>> >>> <pr...@gmail.com> wrote:
>> >>>>
>> >>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being
>> created .
>> >>>> If I do compare with the older path the permissions are same on the
>> parent
>> >>>> directories.
>> >>>> Do I need to create this this directory manually and set the
>> permission
>> >>>> ?
>> >>>>
>> >>>> Older Path
>> >>>>
>> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>> >>>> total 4
>> >>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>> >>>>
>> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>> >>>> total 12
>> >>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>> >>>>
>> >>>>
>> >>>> New Path
>> >>>>
>> >>>> # ll /mnt/san1/hdfs/cache/hdfs/
>> >>>> total 4
>> >>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>> >>>>
>> >>>>
>> >>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>> >>>> total 4
>> >>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>> >>>>
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> The exact error is displayed on your log and should be somewhat self
>> >>>>> explanatory:
>> >>>>>
>> >>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> >>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>> >>>>> state: storage directory does not exist or is not accessible.
>> >>>>>
>> >>>>> Please check this one's availability, permissions (the NN user
>> should
>> >>>>> be able to access it).
>> >>>>>
>> >>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>> >>>>> <pr...@gmail.com> wrote:
>> >>>>> > Please find below logs for shutting down the namenode service. Can
>> >>>>> > anybody
>> >>>>> > check this
>> >>>>> >
>> >>>>> > 2013-02-28 02:07:51,752 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> >>>>> > configuration files. Please update hdfs configuration.
>> >>>>> > 2013-02-28 02:07:51,754 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> >>>>> > configuration files. Please update hdfs configuration.
>> >>>>> > 2013-02-28 02:07:51,754 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> image
>> >>>>> > storage
>> >>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss
>> due
>> >>>>> > to lack
>> >>>>> > of redundant storage directories!
>> >>>>> > 2013-02-28 02:07:51,754 WARN
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>> >>>>> > namespace
>> >>>>> > edits storage directory (dfs.namenode.edits.dir) configured.
>> Beware
>> >>>>> > of
>> >>>>> > dataloss due to lack of redundant storage directories!
>> >>>>> > 2013-02-28 02:07:51,884 INFO
>> org.apache.hadoop.util.HostsFileReader:
>> >>>>> > Refreshing hosts (include/exclude) list
>> >>>>> > 2013-02-28 02:07:51,890 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> >>>>> > dfs.block.invalidate.limit=1000
>> >>>>> > 2013-02-28 02:07:51,909 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > dfs.block.access.token.enable=false
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > defaultReplication = 1
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > maxReplication
>> >>>>> > = 512
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > minReplication
>> >>>>> > = 1
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > maxReplicationStreams = 2
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > shouldCheckForEnoughRacks = false
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > replicationRecheckInterval = 3000
>> >>>>> > 2013-02-28 02:07:51,910 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> >>>>> > encryptDataTransfer = false
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>> >>>>> > =
>> >>>>> > hdfs (auth:SIMPLE)
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>> >>>>> > =
>> >>>>> > hadmin
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > isPermissionEnabled =
>> >>>>> > true
>> >>>>> > 2013-02-28 02:07:51,920 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
>> >>>>> > false
>> >>>>> > 2013-02-28 02:07:51,925 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
>> Enabled:
>> >>>>> > true
>> >>>>> > 2013-02-28 02:07:52,462 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file
>> names
>> >>>>> > occuring
>> >>>>> > more than 10 times
>> >>>>> > 2013-02-28 02:07:52,466 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> >>>>> > 2013-02-28 02:07:52,467 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > dfs.namenode.safemode.min.datanodes = 0
>> >>>>> > 2013-02-28 02:07:52,467 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> >>>>> > dfs.namenode.safemode.extension = 0
>> >>>>> > 2013-02-28 02:07:52,469 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.common.Storage:
>> >>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not
>> exist.
>> >>>>> > 2013-02-28 02:07:52,471 INFO
>> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping
>> NameNode
>> >>>>> > metrics
>> >>>>> > system...
>> >>>>> > 2013-02-28 02:07:52,472 INFO
>> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
>> metrics
>> >>>>> > system
>> >>>>> > stopped.
>> >>>>> > 2013-02-28 02:07:52,473 INFO
>> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
>> metrics
>> >>>>> > system
>> >>>>> > shutdown complete.
>> >>>>> > 2013-02-28 02:07:52,473 FATAL
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>> >>>>> > namenode join
>> >>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> >>>>> > Directory
>> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
>> >>>>> > storage
>> >>>>> > directory does not exist or is not accessible.
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> >>>>> > at
>> >>>>> >
>> >>>>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> >>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil:
>> Exiting
>> >>>>> > with
>> >>>>> > status 1
>> >>>>> > 2013-02-28 02:08:48,913 INFO
>> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> >>>>> > /************************************************************
>> >>>>> > SHUTDOWN_MSG: Shutting down NameNode at
>> >>>>> > OPERA-MAST1.ny.os.local/192.168.1.3
>> >>>>> >
>> >>>>> >
>> >>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>> >>>>> > <pr...@gmail.com> wrote:
>> >>>>> >>
>> >>>>> >> Hi Guys,
>> >>>>> >>
>> >>>>> >> I have space on other partition. Can I change the path for cache
>> >>>>> >> files on
>> >>>>> >> other partition ? I have below properties . Can it resolve the
>> issue
>> >>>>> >> ? If i
>> >>>>> >> change the path to other directories and restart services I get
>> the
>> >>>>> >> below
>> >>>>> >> error while starting the service namenode. I didn't find
>> anything in
>> >>>>> >> logs so
>> >>>>> >> far. Can you please suggest something ?
>> >>>>> >>
>> >>>>> >> <property>
>> >>>>> >> <name>hadoop.tmp.dir</name>
>> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>> >>>>> >> </property>
>> >>>>> >> <property>
>> >>>>> >> <name>dfs.namenode.name.dir</name>
>> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/name</value>
>> >>>>> >> </property>
>> >>>>> >> <property>
>> >>>>> >> <name>dfs.namenode.checkpoint.dir</name>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/namesecondary</value>
>> >>>>> >> </property>
>> >>>>> >> <property>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> Service namenode is failing
>> >>>>> >>
>> >>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
>> >>>>> >> status; done
>> >>>>> >> Hadoop datanode is running [ OK
>> ]
>> >>>>> >> Hadoop namenode is dead and pid file exists
>> [FAILED]
>> >>>>> >> Hadoop secondarynamenode is running [ OK
>> ]
>> >>>>> >>
>> >>>>> >> Thanks,
>> >>>>> >>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>> >>>>> >> <pr...@gmail.com> wrote:
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>> >>>>> >>> wrote:
>> >>>>> >>>>
>> >>>>> >>>> http://NNHOST:50070/conf
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>> Harsh, I changed the value as said & restarted service NN. For
>> >>>>> >>> verifying
>> >>>>> >>> i checked the http link that you gave and i saw the property
>> their
>> >>>>> >>> but on
>> >>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>> >>>>> >>> missing
>> >>>>> >>> blocks. Please check the logs or run fsck in order to identify
>> the
>> >>>>> >>> missing
>> >>>>> >>> blocks.) when i clicked on this link i can see file names .
>> Do I
>> >>>>> >>> need to
>> >>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>> >>>>> >>> command fsck
>> >>>>> >>> that i can run on the running hadoop ?
>> >>>>> >>>
>> >>>>> >>> Thanks,
>> >>>>> >>>
>> >>>>> >>
>> >>>>> >
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Harsh J
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Harsh J
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
thanks Harsh but you didn't answer on this before, I will try to move old
directory "name" to new location and restart services . Hope it will not
loose any data .
old Location
$ sudo ls -l /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 namesecondary
New location
$ sudo ls -l /mnt/san1/hdfs/hdfs/dfs/
total 8
drwx------. 3 hdfs hdfs 4096 Feb 28 11:28 data
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 11:28 namesecondary
Thanks,
On Fri, Mar 1, 2013 at 12:14 PM, Harsh J <ha...@cloudera.com> wrote:
> I believe I already responded to another one of your multiple threads
> on this problem, that you have to move contents of the old dir to the
> new dir and not format. Thats how one changes the dfs.name.dir
> properly.
>
> On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
> <pr...@gmail.com> wrote:
> > Hi,
> >
> > May i know the answer plz ?
> >
> > Thanks,
> >
> >
> > On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
> > <pr...@gmail.com> wrote:
> >>
> >> Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
> >> and set permission . when i restart hadoop services. It removes and I
> find
> >> below logs.
> >>
> >> Do I need to format the NN?
> >> Below is the command to format the NN ?
> >> Any kind of loss while formatting ?
> >> Is there any way to avoid formatting and change the cache path ?
> >>
> >> 2013-02-28 05:57:50,902 INFO
> org.apache.hadoop.hdfs.server.common.Storage:
> >> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by
> nodename
> >> 81133@OPERA-MAST1.ny.os.local
> >> 2013-02-28 05:57:50,904 INFO
> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics
> >> system...
> >> 2013-02-28 05:57:50,904 INFO
> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >> stopped.
> >> 2013-02-28 05:57:50,904 INFO
> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >> shutdown complete.
> >> 2013-02-28 05:57:50,905 FATAL
> >> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
> join
> >> java.io.IOException: NameNode is not formatted.
> >>
> >> Command to format the NN.
> >>
> >> sudo -u hdfs hdfs namenode -format
> >>
> >> Thanks,
> >>
> >>
> >> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
> >> <pr...@gmail.com> wrote:
> >>>
> >>> After creating the directory and setting permission I tried to restart
> >>> the services and i get error
> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
> >>> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are
> not
> >>> being started.
> >>>
> >>> Need to check few logs from below logs.
> >>> ===================================
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> storage
> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
> to lack
> >>> of redundant sto
> >>> rage directories!
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> >>> dataloss due to lack of re
> >>> dundant storage directories!
> >>>
> >>>
> >>> ************************************************************/
> >>> 2013-02-28 05:06:23,385 WARN
> >>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate
> configuration:
> >>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
> >>> 2013-02-28 05:06:23,556 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period
> >>> at 10 second(s).
> >>> 2013-02-28 05:06:23,556 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >>> started
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> storage
> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
> to lack
> >>> of redundant sto
> >>> rage directories!
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> >>> dataloss due to lack of re
> >>> dundant storage directories!
> >>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
> >>> Refreshing hosts (include/exclude) list
> >>> 2013-02-28 05:06:25,623 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> >>> dfs.block.invalidate.limit=1000
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> dfs.block.access.token.enable=false
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> defaultReplication = 1
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication
> >>> = 512
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication
> >>> = 1
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> maxReplicationStreams = 2
> >>> 2013-02-28 05:06:26,016 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> shouldCheckForEnoughRacks = false
> >>> 2013-02-28 05:06:26,016 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> replicationRecheckInterval = 3000
> >>> 2013-02-28 05:06:26,016 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> encryptDataTransfer = false
> >>> 2013-02-28 05:06:26,022 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
> =
> >>> hdfs (auth:SIMPLE)
> >>> 2013-02-28 05:06:26,022 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
> =
> >>> hadmin
> >>> 2013-02-28 05:06:26,022 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled =
> >>> true
> >>> 2013-02-28 05:06:26,023 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> >>> 2013-02-28 05:06:26,026 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
> true
> >>> 2013-02-28 05:06:26,359 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring
> >>> more than 10 times
> >>> 2013-02-28 05:06:26,361 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> >>> 2013-02-28 05:06:26,361 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>> dfs.namenode.safemode.min.datanodes = 0
> >>> 2013-02-28 05:06:26,361 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>> dfs.namenode.safemode.extension = 0
> >>> 2013-02-28 05:06:26,378 INFO
> >>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
> >>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
> >>> 7275@OPERA-MAST1.ny.os.local
> >>> 2013-02-28 05:06:26,381 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics
> >>> system...
> >>> 2013-02-28 05:06:26,381 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >>> stopped.
> >>> 2013-02-28 05:06:26,381 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >>> shutdown complete.
> >>> 2013-02-28 05:06:26,382 FATAL
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
> join
> >>> java.io.IOException: NameNode is not formatted.
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> >>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
> >>> with status 1
> >>> 2013-02-28 05:06:26,394 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> >>> /************************************************************
> >>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
> >>>
> >>>
> >>>
> >>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
> >>> <pr...@gmail.com> wrote:
> >>>>
> >>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being
> created .
> >>>> If I do compare with the older path the permissions are same on the
> parent
> >>>> directories.
> >>>> Do I need to create this this directory manually and set the
> permission
> >>>> ?
> >>>>
> >>>> Older Path
> >>>>
> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
> >>>> total 4
> >>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
> >>>>
> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
> >>>> total 12
> >>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
> >>>>
> >>>>
> >>>> New Path
> >>>>
> >>>> # ll /mnt/san1/hdfs/cache/hdfs/
> >>>> total 4
> >>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
> >>>>
> >>>>
> >>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
> >>>> total 4
> >>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
> >>>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> The exact error is displayed on your log and should be somewhat self
> >>>>> explanatory:
> >>>>>
> >>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> >>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
> >>>>> state: storage directory does not exist or is not accessible.
> >>>>>
> >>>>> Please check this one's availability, permissions (the NN user should
> >>>>> be able to access it).
> >>>>>
> >>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
> >>>>> <pr...@gmail.com> wrote:
> >>>>> > Please find below logs for shutting down the namenode service. Can
> >>>>> > anybody
> >>>>> > check this
> >>>>> >
> >>>>> > 2013-02-28 02:07:51,752 WARN
> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>>>> > configuration files. Please update hdfs configuration.
> >>>>> > 2013-02-28 02:07:51,754 WARN
> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>>>> > configuration files. Please update hdfs configuration.
> >>>>> > 2013-02-28 02:07:51,754 WARN
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> >>>>> > storage
> >>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss
> due
> >>>>> > to lack
> >>>>> > of redundant storage directories!
> >>>>> > 2013-02-28 02:07:51,754 WARN
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
> >>>>> > namespace
> >>>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware
> >>>>> > of
> >>>>> > dataloss due to lack of redundant storage directories!
> >>>>> > 2013-02-28 02:07:51,884 INFO
> org.apache.hadoop.util.HostsFileReader:
> >>>>> > Refreshing hosts (include/exclude) list
> >>>>> > 2013-02-28 02:07:51,890 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> >>>>> > dfs.block.invalidate.limit=1000
> >>>>> > 2013-02-28 02:07:51,909 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > dfs.block.access.token.enable=false
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > defaultReplication = 1
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > maxReplication
> >>>>> > = 512
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > minReplication
> >>>>> > = 1
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > maxReplicationStreams = 2
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > shouldCheckForEnoughRacks = false
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > replicationRecheckInterval = 3000
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > encryptDataTransfer = false
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
> >>>>> > =
> >>>>> > hdfs (auth:SIMPLE)
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
> >>>>> > =
> >>>>> > hadmin
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > isPermissionEnabled =
> >>>>> > true
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
> >>>>> > false
> >>>>> > 2013-02-28 02:07:51,925 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
> Enabled:
> >>>>> > true
> >>>>> > 2013-02-28 02:07:52,462 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> >>>>> > occuring
> >>>>> > more than 10 times
> >>>>> > 2013-02-28 02:07:52,466 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> >>>>> > 2013-02-28 02:07:52,467 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > dfs.namenode.safemode.min.datanodes = 0
> >>>>> > 2013-02-28 02:07:52,467 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > dfs.namenode.safemode.extension = 0
> >>>>> > 2013-02-28 02:07:52,469 INFO
> >>>>> > org.apache.hadoop.hdfs.server.common.Storage:
> >>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not
> exist.
> >>>>> > 2013-02-28 02:07:52,471 INFO
> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping
> NameNode
> >>>>> > metrics
> >>>>> > system...
> >>>>> > 2013-02-28 02:07:52,472 INFO
> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> >>>>> > system
> >>>>> > stopped.
> >>>>> > 2013-02-28 02:07:52,473 INFO
> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> >>>>> > system
> >>>>> > shutdown complete.
> >>>>> > 2013-02-28 02:07:52,473 FATAL
> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
> >>>>> > namenode join
> >>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> >>>>> > Directory
> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
> >>>>> > storage
> >>>>> > directory does not exist or is not accessible.
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> >>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil:
> Exiting
> >>>>> > with
> >>>>> > status 1
> >>>>> > 2013-02-28 02:08:48,913 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> >>>>> > /************************************************************
> >>>>> > SHUTDOWN_MSG: Shutting down NameNode at
> >>>>> > OPERA-MAST1.ny.os.local/192.168.1.3
> >>>>> >
> >>>>> >
> >>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
> >>>>> > <pr...@gmail.com> wrote:
> >>>>> >>
> >>>>> >> Hi Guys,
> >>>>> >>
> >>>>> >> I have space on other partition. Can I change the path for cache
> >>>>> >> files on
> >>>>> >> other partition ? I have below properties . Can it resolve the
> issue
> >>>>> >> ? If i
> >>>>> >> change the path to other directories and restart services I get
> the
> >>>>> >> below
> >>>>> >> error while starting the service namenode. I didn't find anything
> in
> >>>>> >> logs so
> >>>>> >> far. Can you please suggest something ?
> >>>>> >>
> >>>>> >> <property>
> >>>>> >> <name>hadoop.tmp.dir</name>
> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> >>>>> >> </property>
> >>>>> >> <property>
> >>>>> >> <name>dfs.namenode.name.dir</name>
> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/name</value>
> >>>>> >> </property>
> >>>>> >> <property>
> >>>>> >> <name>dfs.namenode.checkpoint.dir</name>
> >>>>> >>
> >>>>> >>
> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/namesecondary</value>
> >>>>> >> </property>
> >>>>> >> <property>
> >>>>> >>
> >>>>> >>
> >>>>> >> Service namenode is failing
> >>>>> >>
> >>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
> >>>>> >> status; done
> >>>>> >> Hadoop datanode is running [ OK
> ]
> >>>>> >> Hadoop namenode is dead and pid file exists
> [FAILED]
> >>>>> >> Hadoop secondarynamenode is running [ OK
> ]
> >>>>> >>
> >>>>> >> Thanks,
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
> >>>>> >> <pr...@gmail.com> wrote:
> >>>>> >>>
> >>>>> >>>
> >>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
> >>>>> >>> wrote:
> >>>>> >>>>
> >>>>> >>>> http://NNHOST:50070/conf
> >>>>> >>>
> >>>>> >>>
> >>>>> >>>
> >>>>> >>> Harsh, I changed the value as said & restarted service NN. For
> >>>>> >>> verifying
> >>>>> >>> i checked the http link that you gave and i saw the property
> their
> >>>>> >>> but on
> >>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
> >>>>> >>> missing
> >>>>> >>> blocks. Please check the logs or run fsck in order to identify
> the
> >>>>> >>> missing
> >>>>> >>> blocks.) when i clicked on this link i can see file names . Do
> I
> >>>>> >>> need to
> >>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
> >>>>> >>> command fsck
> >>>>> >>> that i can run on the running hadoop ?
> >>>>> >>>
> >>>>> >>> Thanks,
> >>>>> >>>
> >>>>> >>
> >>>>> >
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Harsh J
> >>>>
> >>>>
> >>>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
thanks Harsh but you didn't answer on this before, I will try to move old
directory "name" to new location and restart services . Hope it will not
loose any data .
old Location
$ sudo ls -l /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 namesecondary
New location
$ sudo ls -l /mnt/san1/hdfs/hdfs/dfs/
total 8
drwx------. 3 hdfs hdfs 4096 Feb 28 11:28 data
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 11:28 namesecondary
Thanks,
On Fri, Mar 1, 2013 at 12:14 PM, Harsh J <ha...@cloudera.com> wrote:
> I believe I already responded to another one of your multiple threads
> on this problem, that you have to move contents of the old dir to the
> new dir and not format. Thats how one changes the dfs.name.dir
> properly.
>
> On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
> <pr...@gmail.com> wrote:
> > Hi,
> >
> > May i know the answer plz ?
> >
> > Thanks,
> >
> >
> > On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
> > <pr...@gmail.com> wrote:
> >>
> >> Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
> >> and set permission . when i restart hadoop services. It removes and I
> find
> >> below logs.
> >>
> >> Do I need to format the NN?
> >> Below is the command to format the NN ?
> >> Any kind of loss while formatting ?
> >> Is there any way to avoid formatting and change the cache path ?
> >>
> >> 2013-02-28 05:57:50,902 INFO
> org.apache.hadoop.hdfs.server.common.Storage:
> >> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by
> nodename
> >> 81133@OPERA-MAST1.ny.os.local
> >> 2013-02-28 05:57:50,904 INFO
> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics
> >> system...
> >> 2013-02-28 05:57:50,904 INFO
> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >> stopped.
> >> 2013-02-28 05:57:50,904 INFO
> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >> shutdown complete.
> >> 2013-02-28 05:57:50,905 FATAL
> >> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
> join
> >> java.io.IOException: NameNode is not formatted.
> >>
> >> Command to format the NN.
> >>
> >> sudo -u hdfs hdfs namenode -format
> >>
> >> Thanks,
> >>
> >>
> >> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
> >> <pr...@gmail.com> wrote:
> >>>
> >>> After creating the directory and setting permission I tried to restart
> >>> the services and i get error
> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
> >>> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are
> not
> >>> being started.
> >>>
> >>> Need to check few logs from below logs.
> >>> ===================================
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> storage
> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
> to lack
> >>> of redundant sto
> >>> rage directories!
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> >>> dataloss due to lack of re
> >>> dundant storage directories!
> >>>
> >>>
> >>> ************************************************************/
> >>> 2013-02-28 05:06:23,385 WARN
> >>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate
> configuration:
> >>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
> >>> 2013-02-28 05:06:23,556 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period
> >>> at 10 second(s).
> >>> 2013-02-28 05:06:23,556 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >>> started
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> storage
> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
> to lack
> >>> of redundant sto
> >>> rage directories!
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> >>> dataloss due to lack of re
> >>> dundant storage directories!
> >>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
> >>> Refreshing hosts (include/exclude) list
> >>> 2013-02-28 05:06:25,623 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> >>> dfs.block.invalidate.limit=1000
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> dfs.block.access.token.enable=false
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> defaultReplication = 1
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication
> >>> = 512
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication
> >>> = 1
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> maxReplicationStreams = 2
> >>> 2013-02-28 05:06:26,016 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> shouldCheckForEnoughRacks = false
> >>> 2013-02-28 05:06:26,016 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> replicationRecheckInterval = 3000
> >>> 2013-02-28 05:06:26,016 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> encryptDataTransfer = false
> >>> 2013-02-28 05:06:26,022 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
> =
> >>> hdfs (auth:SIMPLE)
> >>> 2013-02-28 05:06:26,022 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
> =
> >>> hadmin
> >>> 2013-02-28 05:06:26,022 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled =
> >>> true
> >>> 2013-02-28 05:06:26,023 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> >>> 2013-02-28 05:06:26,026 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
> true
> >>> 2013-02-28 05:06:26,359 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring
> >>> more than 10 times
> >>> 2013-02-28 05:06:26,361 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> >>> 2013-02-28 05:06:26,361 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>> dfs.namenode.safemode.min.datanodes = 0
> >>> 2013-02-28 05:06:26,361 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>> dfs.namenode.safemode.extension = 0
> >>> 2013-02-28 05:06:26,378 INFO
> >>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
> >>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
> >>> 7275@OPERA-MAST1.ny.os.local
> >>> 2013-02-28 05:06:26,381 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics
> >>> system...
> >>> 2013-02-28 05:06:26,381 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >>> stopped.
> >>> 2013-02-28 05:06:26,381 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >>> shutdown complete.
> >>> 2013-02-28 05:06:26,382 FATAL
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
> join
> >>> java.io.IOException: NameNode is not formatted.
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> >>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
> >>> with status 1
> >>> 2013-02-28 05:06:26,394 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> >>> /************************************************************
> >>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
> >>>
> >>>
> >>>
> >>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
> >>> <pr...@gmail.com> wrote:
> >>>>
> >>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being
> created .
> >>>> If I do compare with the older path the permissions are same on the
> parent
> >>>> directories.
> >>>> Do I need to create this this directory manually and set the
> permission
> >>>> ?
> >>>>
> >>>> Older Path
> >>>>
> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
> >>>> total 4
> >>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
> >>>>
> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
> >>>> total 12
> >>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
> >>>>
> >>>>
> >>>> New Path
> >>>>
> >>>> # ll /mnt/san1/hdfs/cache/hdfs/
> >>>> total 4
> >>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
> >>>>
> >>>>
> >>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
> >>>> total 4
> >>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
> >>>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> The exact error is displayed on your log and should be somewhat self
> >>>>> explanatory:
> >>>>>
> >>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> >>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
> >>>>> state: storage directory does not exist or is not accessible.
> >>>>>
> >>>>> Please check this one's availability, permissions (the NN user should
> >>>>> be able to access it).
> >>>>>
> >>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
> >>>>> <pr...@gmail.com> wrote:
> >>>>> > Please find below logs for shutting down the namenode service. Can
> >>>>> > anybody
> >>>>> > check this
> >>>>> >
> >>>>> > 2013-02-28 02:07:51,752 WARN
> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>>>> > configuration files. Please update hdfs configuration.
> >>>>> > 2013-02-28 02:07:51,754 WARN
> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>>>> > configuration files. Please update hdfs configuration.
> >>>>> > 2013-02-28 02:07:51,754 WARN
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> >>>>> > storage
> >>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss
> due
> >>>>> > to lack
> >>>>> > of redundant storage directories!
> >>>>> > 2013-02-28 02:07:51,754 WARN
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
> >>>>> > namespace
> >>>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware
> >>>>> > of
> >>>>> > dataloss due to lack of redundant storage directories!
> >>>>> > 2013-02-28 02:07:51,884 INFO
> org.apache.hadoop.util.HostsFileReader:
> >>>>> > Refreshing hosts (include/exclude) list
> >>>>> > 2013-02-28 02:07:51,890 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> >>>>> > dfs.block.invalidate.limit=1000
> >>>>> > 2013-02-28 02:07:51,909 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > dfs.block.access.token.enable=false
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > defaultReplication = 1
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > maxReplication
> >>>>> > = 512
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > minReplication
> >>>>> > = 1
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > maxReplicationStreams = 2
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > shouldCheckForEnoughRacks = false
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > replicationRecheckInterval = 3000
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > encryptDataTransfer = false
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
> >>>>> > =
> >>>>> > hdfs (auth:SIMPLE)
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
> >>>>> > =
> >>>>> > hadmin
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > isPermissionEnabled =
> >>>>> > true
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
> >>>>> > false
> >>>>> > 2013-02-28 02:07:51,925 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
> Enabled:
> >>>>> > true
> >>>>> > 2013-02-28 02:07:52,462 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> >>>>> > occuring
> >>>>> > more than 10 times
> >>>>> > 2013-02-28 02:07:52,466 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> >>>>> > 2013-02-28 02:07:52,467 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > dfs.namenode.safemode.min.datanodes = 0
> >>>>> > 2013-02-28 02:07:52,467 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > dfs.namenode.safemode.extension = 0
> >>>>> > 2013-02-28 02:07:52,469 INFO
> >>>>> > org.apache.hadoop.hdfs.server.common.Storage:
> >>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not
> exist.
> >>>>> > 2013-02-28 02:07:52,471 INFO
> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping
> NameNode
> >>>>> > metrics
> >>>>> > system...
> >>>>> > 2013-02-28 02:07:52,472 INFO
> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> >>>>> > system
> >>>>> > stopped.
> >>>>> > 2013-02-28 02:07:52,473 INFO
> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> >>>>> > system
> >>>>> > shutdown complete.
> >>>>> > 2013-02-28 02:07:52,473 FATAL
> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
> >>>>> > namenode join
> >>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> >>>>> > Directory
> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
> >>>>> > storage
> >>>>> > directory does not exist or is not accessible.
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> >>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil:
> Exiting
> >>>>> > with
> >>>>> > status 1
> >>>>> > 2013-02-28 02:08:48,913 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> >>>>> > /************************************************************
> >>>>> > SHUTDOWN_MSG: Shutting down NameNode at
> >>>>> > OPERA-MAST1.ny.os.local/192.168.1.3
> >>>>> >
> >>>>> >
> >>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
> >>>>> > <pr...@gmail.com> wrote:
> >>>>> >>
> >>>>> >> Hi Guys,
> >>>>> >>
> >>>>> >> I have space on other partition. Can I change the path for cache
> >>>>> >> files on
> >>>>> >> other partition ? I have below properties . Can it resolve the
> issue
> >>>>> >> ? If i
> >>>>> >> change the path to other directories and restart services I get
> the
> >>>>> >> below
> >>>>> >> error while starting the service namenode. I didn't find anything
> in
> >>>>> >> logs so
> >>>>> >> far. Can you please suggest something ?
> >>>>> >>
> >>>>> >> <property>
> >>>>> >> <name>hadoop.tmp.dir</name>
> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> >>>>> >> </property>
> >>>>> >> <property>
> >>>>> >> <name>dfs.namenode.name.dir</name>
> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/name</value>
> >>>>> >> </property>
> >>>>> >> <property>
> >>>>> >> <name>dfs.namenode.checkpoint.dir</name>
> >>>>> >>
> >>>>> >>
> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/namesecondary</value>
> >>>>> >> </property>
> >>>>> >> <property>
> >>>>> >>
> >>>>> >>
> >>>>> >> Service namenode is failing
> >>>>> >>
> >>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
> >>>>> >> status; done
> >>>>> >> Hadoop datanode is running [ OK
> ]
> >>>>> >> Hadoop namenode is dead and pid file exists
> [FAILED]
> >>>>> >> Hadoop secondarynamenode is running [ OK
> ]
> >>>>> >>
> >>>>> >> Thanks,
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
> >>>>> >> <pr...@gmail.com> wrote:
> >>>>> >>>
> >>>>> >>>
> >>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
> >>>>> >>> wrote:
> >>>>> >>>>
> >>>>> >>>> http://NNHOST:50070/conf
> >>>>> >>>
> >>>>> >>>
> >>>>> >>>
> >>>>> >>> Harsh, I changed the value as said & restarted service NN. For
> >>>>> >>> verifying
> >>>>> >>> i checked the http link that you gave and i saw the property
> their
> >>>>> >>> but on
> >>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
> >>>>> >>> missing
> >>>>> >>> blocks. Please check the logs or run fsck in order to identify
> the
> >>>>> >>> missing
> >>>>> >>> blocks.) when i clicked on this link i can see file names . Do
> I
> >>>>> >>> need to
> >>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
> >>>>> >>> command fsck
> >>>>> >>> that i can run on the running hadoop ?
> >>>>> >>>
> >>>>> >>> Thanks,
> >>>>> >>>
> >>>>> >>
> >>>>> >
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Harsh J
> >>>>
> >>>>
> >>>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
thanks Harsh but you didn't answer on this before, I will try to move old
directory "name" to new location and restart services . Hope it will not
loose any data .
old Location
$ sudo ls -l /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 namesecondary
New location
$ sudo ls -l /mnt/san1/hdfs/hdfs/dfs/
total 8
drwx------. 3 hdfs hdfs 4096 Feb 28 11:28 data
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 11:28 namesecondary
Thanks,
On Fri, Mar 1, 2013 at 12:14 PM, Harsh J <ha...@cloudera.com> wrote:
> I believe I already responded to another one of your multiple threads
> on this problem, that you have to move contents of the old dir to the
> new dir and not format. Thats how one changes the dfs.name.dir
> properly.
>
> On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
> <pr...@gmail.com> wrote:
> > Hi,
> >
> > May i know the answer plz ?
> >
> > Thanks,
> >
> >
> > On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
> > <pr...@gmail.com> wrote:
> >>
> >> Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
> >> and set permission . when i restart hadoop services. It removes and I
> find
> >> below logs.
> >>
> >> Do I need to format the NN?
> >> Below is the command to format the NN ?
> >> Any kind of loss while formatting ?
> >> Is there any way to avoid formatting and change the cache path ?
> >>
> >> 2013-02-28 05:57:50,902 INFO
> org.apache.hadoop.hdfs.server.common.Storage:
> >> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by
> nodename
> >> 81133@OPERA-MAST1.ny.os.local
> >> 2013-02-28 05:57:50,904 INFO
> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics
> >> system...
> >> 2013-02-28 05:57:50,904 INFO
> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >> stopped.
> >> 2013-02-28 05:57:50,904 INFO
> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >> shutdown complete.
> >> 2013-02-28 05:57:50,905 FATAL
> >> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
> join
> >> java.io.IOException: NameNode is not formatted.
> >>
> >> Command to format the NN.
> >>
> >> sudo -u hdfs hdfs namenode -format
> >>
> >> Thanks,
> >>
> >>
> >> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
> >> <pr...@gmail.com> wrote:
> >>>
> >>> After creating the directory and setting permission I tried to restart
> >>> the services and i get error
> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
> >>> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are
> not
> >>> being started.
> >>>
> >>> Need to check few logs from below logs.
> >>> ===================================
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> storage
> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
> to lack
> >>> of redundant sto
> >>> rage directories!
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> >>> dataloss due to lack of re
> >>> dundant storage directories!
> >>>
> >>>
> >>> ************************************************************/
> >>> 2013-02-28 05:06:23,385 WARN
> >>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate
> configuration:
> >>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
> >>> 2013-02-28 05:06:23,556 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period
> >>> at 10 second(s).
> >>> 2013-02-28 05:06:23,556 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >>> started
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> storage
> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
> to lack
> >>> of redundant sto
> >>> rage directories!
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> >>> dataloss due to lack of re
> >>> dundant storage directories!
> >>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
> >>> Refreshing hosts (include/exclude) list
> >>> 2013-02-28 05:06:25,623 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> >>> dfs.block.invalidate.limit=1000
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> dfs.block.access.token.enable=false
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> defaultReplication = 1
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication
> >>> = 512
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication
> >>> = 1
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> maxReplicationStreams = 2
> >>> 2013-02-28 05:06:26,016 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> shouldCheckForEnoughRacks = false
> >>> 2013-02-28 05:06:26,016 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> replicationRecheckInterval = 3000
> >>> 2013-02-28 05:06:26,016 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> encryptDataTransfer = false
> >>> 2013-02-28 05:06:26,022 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
> =
> >>> hdfs (auth:SIMPLE)
> >>> 2013-02-28 05:06:26,022 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
> =
> >>> hadmin
> >>> 2013-02-28 05:06:26,022 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled =
> >>> true
> >>> 2013-02-28 05:06:26,023 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> >>> 2013-02-28 05:06:26,026 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
> true
> >>> 2013-02-28 05:06:26,359 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring
> >>> more than 10 times
> >>> 2013-02-28 05:06:26,361 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> >>> 2013-02-28 05:06:26,361 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>> dfs.namenode.safemode.min.datanodes = 0
> >>> 2013-02-28 05:06:26,361 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>> dfs.namenode.safemode.extension = 0
> >>> 2013-02-28 05:06:26,378 INFO
> >>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
> >>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
> >>> 7275@OPERA-MAST1.ny.os.local
> >>> 2013-02-28 05:06:26,381 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics
> >>> system...
> >>> 2013-02-28 05:06:26,381 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >>> stopped.
> >>> 2013-02-28 05:06:26,381 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >>> shutdown complete.
> >>> 2013-02-28 05:06:26,382 FATAL
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
> join
> >>> java.io.IOException: NameNode is not formatted.
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> >>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
> >>> with status 1
> >>> 2013-02-28 05:06:26,394 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> >>> /************************************************************
> >>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
> >>>
> >>>
> >>>
> >>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
> >>> <pr...@gmail.com> wrote:
> >>>>
> >>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being
> created .
> >>>> If I do compare with the older path the permissions are same on the
> parent
> >>>> directories.
> >>>> Do I need to create this this directory manually and set the
> permission
> >>>> ?
> >>>>
> >>>> Older Path
> >>>>
> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
> >>>> total 4
> >>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
> >>>>
> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
> >>>> total 12
> >>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
> >>>>
> >>>>
> >>>> New Path
> >>>>
> >>>> # ll /mnt/san1/hdfs/cache/hdfs/
> >>>> total 4
> >>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
> >>>>
> >>>>
> >>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
> >>>> total 4
> >>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
> >>>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> The exact error is displayed on your log and should be somewhat self
> >>>>> explanatory:
> >>>>>
> >>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> >>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
> >>>>> state: storage directory does not exist or is not accessible.
> >>>>>
> >>>>> Please check this one's availability, permissions (the NN user should
> >>>>> be able to access it).
> >>>>>
> >>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
> >>>>> <pr...@gmail.com> wrote:
> >>>>> > Please find below logs for shutting down the namenode service. Can
> >>>>> > anybody
> >>>>> > check this
> >>>>> >
> >>>>> > 2013-02-28 02:07:51,752 WARN
> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>>>> > configuration files. Please update hdfs configuration.
> >>>>> > 2013-02-28 02:07:51,754 WARN
> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>>>> > configuration files. Please update hdfs configuration.
> >>>>> > 2013-02-28 02:07:51,754 WARN
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> >>>>> > storage
> >>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss
> due
> >>>>> > to lack
> >>>>> > of redundant storage directories!
> >>>>> > 2013-02-28 02:07:51,754 WARN
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
> >>>>> > namespace
> >>>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware
> >>>>> > of
> >>>>> > dataloss due to lack of redundant storage directories!
> >>>>> > 2013-02-28 02:07:51,884 INFO
> org.apache.hadoop.util.HostsFileReader:
> >>>>> > Refreshing hosts (include/exclude) list
> >>>>> > 2013-02-28 02:07:51,890 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> >>>>> > dfs.block.invalidate.limit=1000
> >>>>> > 2013-02-28 02:07:51,909 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > dfs.block.access.token.enable=false
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > defaultReplication = 1
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > maxReplication
> >>>>> > = 512
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > minReplication
> >>>>> > = 1
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > maxReplicationStreams = 2
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > shouldCheckForEnoughRacks = false
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > replicationRecheckInterval = 3000
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > encryptDataTransfer = false
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
> >>>>> > =
> >>>>> > hdfs (auth:SIMPLE)
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
> >>>>> > =
> >>>>> > hadmin
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > isPermissionEnabled =
> >>>>> > true
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
> >>>>> > false
> >>>>> > 2013-02-28 02:07:51,925 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
> Enabled:
> >>>>> > true
> >>>>> > 2013-02-28 02:07:52,462 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> >>>>> > occuring
> >>>>> > more than 10 times
> >>>>> > 2013-02-28 02:07:52,466 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> >>>>> > 2013-02-28 02:07:52,467 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > dfs.namenode.safemode.min.datanodes = 0
> >>>>> > 2013-02-28 02:07:52,467 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > dfs.namenode.safemode.extension = 0
> >>>>> > 2013-02-28 02:07:52,469 INFO
> >>>>> > org.apache.hadoop.hdfs.server.common.Storage:
> >>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not
> exist.
> >>>>> > 2013-02-28 02:07:52,471 INFO
> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping
> NameNode
> >>>>> > metrics
> >>>>> > system...
> >>>>> > 2013-02-28 02:07:52,472 INFO
> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> >>>>> > system
> >>>>> > stopped.
> >>>>> > 2013-02-28 02:07:52,473 INFO
> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> >>>>> > system
> >>>>> > shutdown complete.
> >>>>> > 2013-02-28 02:07:52,473 FATAL
> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
> >>>>> > namenode join
> >>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> >>>>> > Directory
> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
> >>>>> > storage
> >>>>> > directory does not exist or is not accessible.
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> >>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil:
> Exiting
> >>>>> > with
> >>>>> > status 1
> >>>>> > 2013-02-28 02:08:48,913 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> >>>>> > /************************************************************
> >>>>> > SHUTDOWN_MSG: Shutting down NameNode at
> >>>>> > OPERA-MAST1.ny.os.local/192.168.1.3
> >>>>> >
> >>>>> >
> >>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
> >>>>> > <pr...@gmail.com> wrote:
> >>>>> >>
> >>>>> >> Hi Guys,
> >>>>> >>
> >>>>> >> I have space on other partition. Can I change the path for cache
> >>>>> >> files on
> >>>>> >> other partition ? I have below properties . Can it resolve the
> issue
> >>>>> >> ? If i
> >>>>> >> change the path to other directories and restart services I get
> the
> >>>>> >> below
> >>>>> >> error while starting the service namenode. I didn't find anything
> in
> >>>>> >> logs so
> >>>>> >> far. Can you please suggest something ?
> >>>>> >>
> >>>>> >> <property>
> >>>>> >> <name>hadoop.tmp.dir</name>
> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> >>>>> >> </property>
> >>>>> >> <property>
> >>>>> >> <name>dfs.namenode.name.dir</name>
> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/name</value>
> >>>>> >> </property>
> >>>>> >> <property>
> >>>>> >> <name>dfs.namenode.checkpoint.dir</name>
> >>>>> >>
> >>>>> >>
> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/namesecondary</value>
> >>>>> >> </property>
> >>>>> >> <property>
> >>>>> >>
> >>>>> >>
> >>>>> >> Service namenode is failing
> >>>>> >>
> >>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
> >>>>> >> status; done
> >>>>> >> Hadoop datanode is running [ OK
> ]
> >>>>> >> Hadoop namenode is dead and pid file exists
> [FAILED]
> >>>>> >> Hadoop secondarynamenode is running [ OK
> ]
> >>>>> >>
> >>>>> >> Thanks,
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
> >>>>> >> <pr...@gmail.com> wrote:
> >>>>> >>>
> >>>>> >>>
> >>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
> >>>>> >>> wrote:
> >>>>> >>>>
> >>>>> >>>> http://NNHOST:50070/conf
> >>>>> >>>
> >>>>> >>>
> >>>>> >>>
> >>>>> >>> Harsh, I changed the value as said & restarted service NN. For
> >>>>> >>> verifying
> >>>>> >>> i checked the http link that you gave and i saw the property
> their
> >>>>> >>> but on
> >>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
> >>>>> >>> missing
> >>>>> >>> blocks. Please check the logs or run fsck in order to identify
> the
> >>>>> >>> missing
> >>>>> >>> blocks.) when i clicked on this link i can see file names . Do
> I
> >>>>> >>> need to
> >>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
> >>>>> >>> command fsck
> >>>>> >>> that i can run on the running hadoop ?
> >>>>> >>>
> >>>>> >>> Thanks,
> >>>>> >>>
> >>>>> >>
> >>>>> >
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Harsh J
> >>>>
> >>>>
> >>>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
thanks Harsh but you didn't answer on this before, I will try to move old
directory "name" to new location and restart services . Hope it will not
loose any data .
old Location
$ sudo ls -l /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 namesecondary
New location
$ sudo ls -l /mnt/san1/hdfs/hdfs/dfs/
total 8
drwx------. 3 hdfs hdfs 4096 Feb 28 11:28 data
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 11:28 namesecondary
Thanks,
On Fri, Mar 1, 2013 at 12:14 PM, Harsh J <ha...@cloudera.com> wrote:
> I believe I already responded to another one of your multiple threads
> on this problem, that you have to move contents of the old dir to the
> new dir and not format. Thats how one changes the dfs.name.dir
> properly.
>
> On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
> <pr...@gmail.com> wrote:
> > Hi,
> >
> > May i know the answer plz ?
> >
> > Thanks,
> >
> >
> > On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
> > <pr...@gmail.com> wrote:
> >>
> >> Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
> >> and set permission . when i restart hadoop services. It removes and I
> find
> >> below logs.
> >>
> >> Do I need to format the NN?
> >> Below is the command to format the NN ?
> >> Any kind of loss while formatting ?
> >> Is there any way to avoid formatting and change the cache path ?
> >>
> >> 2013-02-28 05:57:50,902 INFO
> org.apache.hadoop.hdfs.server.common.Storage:
> >> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by
> nodename
> >> 81133@OPERA-MAST1.ny.os.local
> >> 2013-02-28 05:57:50,904 INFO
> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics
> >> system...
> >> 2013-02-28 05:57:50,904 INFO
> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >> stopped.
> >> 2013-02-28 05:57:50,904 INFO
> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >> shutdown complete.
> >> 2013-02-28 05:57:50,905 FATAL
> >> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
> join
> >> java.io.IOException: NameNode is not formatted.
> >>
> >> Command to format the NN.
> >>
> >> sudo -u hdfs hdfs namenode -format
> >>
> >> Thanks,
> >>
> >>
> >> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
> >> <pr...@gmail.com> wrote:
> >>>
> >>> After creating the directory and setting permission I tried to restart
> >>> the services and i get error
> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
> >>> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are
> not
> >>> being started.
> >>>
> >>> Need to check few logs from below logs.
> >>> ===================================
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> storage
> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
> to lack
> >>> of redundant sto
> >>> rage directories!
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> >>> dataloss due to lack of re
> >>> dundant storage directories!
> >>>
> >>>
> >>> ************************************************************/
> >>> 2013-02-28 05:06:23,385 WARN
> >>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate
> configuration:
> >>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
> >>> 2013-02-28 05:06:23,556 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period
> >>> at 10 second(s).
> >>> 2013-02-28 05:06:23,556 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >>> started
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>> configuration files. Please update hdfs configuration.
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> storage
> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due
> to lack
> >>> of redundant sto
> >>> rage directories!
> >>> 2013-02-28 05:06:24,906 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> >>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> >>> dataloss due to lack of re
> >>> dundant storage directories!
> >>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
> >>> Refreshing hosts (include/exclude) list
> >>> 2013-02-28 05:06:25,623 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> >>> dfs.block.invalidate.limit=1000
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> dfs.block.access.token.enable=false
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> defaultReplication = 1
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication
> >>> = 512
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication
> >>> = 1
> >>> 2013-02-28 05:06:26,015 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> maxReplicationStreams = 2
> >>> 2013-02-28 05:06:26,016 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> shouldCheckForEnoughRacks = false
> >>> 2013-02-28 05:06:26,016 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> replicationRecheckInterval = 3000
> >>> 2013-02-28 05:06:26,016 INFO
> >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>> encryptDataTransfer = false
> >>> 2013-02-28 05:06:26,022 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
> =
> >>> hdfs (auth:SIMPLE)
> >>> 2013-02-28 05:06:26,022 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
> =
> >>> hadmin
> >>> 2013-02-28 05:06:26,022 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled =
> >>> true
> >>> 2013-02-28 05:06:26,023 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> >>> 2013-02-28 05:06:26,026 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
> true
> >>> 2013-02-28 05:06:26,359 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring
> >>> more than 10 times
> >>> 2013-02-28 05:06:26,361 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> >>> 2013-02-28 05:06:26,361 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>> dfs.namenode.safemode.min.datanodes = 0
> >>> 2013-02-28 05:06:26,361 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>> dfs.namenode.safemode.extension = 0
> >>> 2013-02-28 05:06:26,378 INFO
> >>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
> >>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
> >>> 7275@OPERA-MAST1.ny.os.local
> >>> 2013-02-28 05:06:26,381 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics
> >>> system...
> >>> 2013-02-28 05:06:26,381 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >>> stopped.
> >>> 2013-02-28 05:06:26,381 INFO
> >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> >>> shutdown complete.
> >>> 2013-02-28 05:06:26,382 FATAL
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
> join
> >>> java.io.IOException: NameNode is not formatted.
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> >>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
> >>> with status 1
> >>> 2013-02-28 05:06:26,394 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> >>> /************************************************************
> >>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
> >>>
> >>>
> >>>
> >>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
> >>> <pr...@gmail.com> wrote:
> >>>>
> >>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being
> created .
> >>>> If I do compare with the older path the permissions are same on the
> parent
> >>>> directories.
> >>>> Do I need to create this this directory manually and set the
> permission
> >>>> ?
> >>>>
> >>>> Older Path
> >>>>
> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
> >>>> total 4
> >>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
> >>>>
> >>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
> >>>> total 12
> >>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
> >>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
> >>>>
> >>>>
> >>>> New Path
> >>>>
> >>>> # ll /mnt/san1/hdfs/cache/hdfs/
> >>>> total 4
> >>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
> >>>>
> >>>>
> >>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
> >>>> total 4
> >>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
> >>>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> The exact error is displayed on your log and should be somewhat self
> >>>>> explanatory:
> >>>>>
> >>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> >>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
> >>>>> state: storage directory does not exist or is not accessible.
> >>>>>
> >>>>> Please check this one's availability, permissions (the NN user should
> >>>>> be able to access it).
> >>>>>
> >>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
> >>>>> <pr...@gmail.com> wrote:
> >>>>> > Please find below logs for shutting down the namenode service. Can
> >>>>> > anybody
> >>>>> > check this
> >>>>> >
> >>>>> > 2013-02-28 02:07:51,752 WARN
> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>>>> > configuration files. Please update hdfs configuration.
> >>>>> > 2013-02-28 02:07:51,754 WARN
> >>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> >>>>> > configuration files. Please update hdfs configuration.
> >>>>> > 2013-02-28 02:07:51,754 WARN
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> >>>>> > storage
> >>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss
> due
> >>>>> > to lack
> >>>>> > of redundant storage directories!
> >>>>> > 2013-02-28 02:07:51,754 WARN
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
> >>>>> > namespace
> >>>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware
> >>>>> > of
> >>>>> > dataloss due to lack of redundant storage directories!
> >>>>> > 2013-02-28 02:07:51,884 INFO
> org.apache.hadoop.util.HostsFileReader:
> >>>>> > Refreshing hosts (include/exclude) list
> >>>>> > 2013-02-28 02:07:51,890 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> >>>>> > dfs.block.invalidate.limit=1000
> >>>>> > 2013-02-28 02:07:51,909 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > dfs.block.access.token.enable=false
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > defaultReplication = 1
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > maxReplication
> >>>>> > = 512
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > minReplication
> >>>>> > = 1
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > maxReplicationStreams = 2
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > shouldCheckForEnoughRacks = false
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > replicationRecheckInterval = 3000
> >>>>> > 2013-02-28 02:07:51,910 INFO
> >>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> >>>>> > encryptDataTransfer = false
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
> >>>>> > =
> >>>>> > hdfs (auth:SIMPLE)
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
> >>>>> > =
> >>>>> > hadmin
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > isPermissionEnabled =
> >>>>> > true
> >>>>> > 2013-02-28 02:07:51,920 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
> >>>>> > false
> >>>>> > 2013-02-28 02:07:51,925 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
> Enabled:
> >>>>> > true
> >>>>> > 2013-02-28 02:07:52,462 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> >>>>> > occuring
> >>>>> > more than 10 times
> >>>>> > 2013-02-28 02:07:52,466 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> >>>>> > 2013-02-28 02:07:52,467 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > dfs.namenode.safemode.min.datanodes = 0
> >>>>> > 2013-02-28 02:07:52,467 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> >>>>> > dfs.namenode.safemode.extension = 0
> >>>>> > 2013-02-28 02:07:52,469 INFO
> >>>>> > org.apache.hadoop.hdfs.server.common.Storage:
> >>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not
> exist.
> >>>>> > 2013-02-28 02:07:52,471 INFO
> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping
> NameNode
> >>>>> > metrics
> >>>>> > system...
> >>>>> > 2013-02-28 02:07:52,472 INFO
> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> >>>>> > system
> >>>>> > stopped.
> >>>>> > 2013-02-28 02:07:52,473 INFO
> >>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> >>>>> > system
> >>>>> > shutdown complete.
> >>>>> > 2013-02-28 02:07:52,473 FATAL
> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
> >>>>> > namenode join
> >>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> >>>>> > Directory
> >>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
> >>>>> > storage
> >>>>> > directory does not exist or is not accessible.
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> >>>>> > at
> >>>>> >
> >>>>> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> >>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil:
> Exiting
> >>>>> > with
> >>>>> > status 1
> >>>>> > 2013-02-28 02:08:48,913 INFO
> >>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> >>>>> > /************************************************************
> >>>>> > SHUTDOWN_MSG: Shutting down NameNode at
> >>>>> > OPERA-MAST1.ny.os.local/192.168.1.3
> >>>>> >
> >>>>> >
> >>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
> >>>>> > <pr...@gmail.com> wrote:
> >>>>> >>
> >>>>> >> Hi Guys,
> >>>>> >>
> >>>>> >> I have space on other partition. Can I change the path for cache
> >>>>> >> files on
> >>>>> >> other partition ? I have below properties . Can it resolve the
> issue
> >>>>> >> ? If i
> >>>>> >> change the path to other directories and restart services I get
> the
> >>>>> >> below
> >>>>> >> error while starting the service namenode. I didn't find anything
> in
> >>>>> >> logs so
> >>>>> >> far. Can you please suggest something ?
> >>>>> >>
> >>>>> >> <property>
> >>>>> >> <name>hadoop.tmp.dir</name>
> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> >>>>> >> </property>
> >>>>> >> <property>
> >>>>> >> <name>dfs.namenode.name.dir</name>
> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/name</value>
> >>>>> >> </property>
> >>>>> >> <property>
> >>>>> >> <name>dfs.namenode.checkpoint.dir</name>
> >>>>> >>
> >>>>> >>
> >>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/namesecondary</value>
> >>>>> >> </property>
> >>>>> >> <property>
> >>>>> >>
> >>>>> >>
> >>>>> >> Service namenode is failing
> >>>>> >>
> >>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
> >>>>> >> status; done
> >>>>> >> Hadoop datanode is running [ OK
> ]
> >>>>> >> Hadoop namenode is dead and pid file exists
> [FAILED]
> >>>>> >> Hadoop secondarynamenode is running [ OK
> ]
> >>>>> >>
> >>>>> >> Thanks,
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
> >>>>> >> <pr...@gmail.com> wrote:
> >>>>> >>>
> >>>>> >>>
> >>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
> >>>>> >>> wrote:
> >>>>> >>>>
> >>>>> >>>> http://NNHOST:50070/conf
> >>>>> >>>
> >>>>> >>>
> >>>>> >>>
> >>>>> >>> Harsh, I changed the value as said & restarted service NN. For
> >>>>> >>> verifying
> >>>>> >>> i checked the http link that you gave and i saw the property
> their
> >>>>> >>> but on
> >>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
> >>>>> >>> missing
> >>>>> >>> blocks. Please check the logs or run fsck in order to identify
> the
> >>>>> >>> missing
> >>>>> >>> blocks.) when i clicked on this link i can see file names . Do
> I
> >>>>> >>> need to
> >>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
> >>>>> >>> command fsck
> >>>>> >>> that i can run on the running hadoop ?
> >>>>> >>>
> >>>>> >>> Thanks,
> >>>>> >>>
> >>>>> >>
> >>>>> >
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Harsh J
> >>>>
> >>>>
> >>>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: NameNode low on available disk space
Posted by Harsh J <ha...@cloudera.com>.
I believe I already responded to another one of your multiple threads
on this problem, that you have to move contents of the old dir to the
new dir and not format. Thats how one changes the dfs.name.dir
properly.
On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
<pr...@gmail.com> wrote:
> Hi,
>
> May i know the answer plz ?
>
> Thanks,
>
>
> On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
> <pr...@gmail.com> wrote:
>>
>> Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>> and set permission . when i restart hadoop services. It removes and I find
>> below logs.
>>
>> Do I need to format the NN?
>> Below is the command to format the NN ?
>> Any kind of loss while formatting ?
>> Is there any way to avoid formatting and change the cache path ?
>>
>> 2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
>> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> 81133@OPERA-MAST1.ny.os.local
>> 2013-02-28 05:57:50,904 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
>> system...
>> 2013-02-28 05:57:50,904 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> stopped.
>> 2013-02-28 05:57:50,904 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> shutdown complete.
>> 2013-02-28 05:57:50,905 FATAL
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>> java.io.IOException: NameNode is not formatted.
>>
>> Command to format the NN.
>>
>> sudo -u hdfs hdfs namenode -format
>>
>> Thanks,
>>
>>
>> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
>> <pr...@gmail.com> wrote:
>>>
>>> After creating the directory and setting permission I tried to restart
>>> the services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>>> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are not
>>> being started.
>>>
>>> Need to check few logs from below logs.
>>> ===================================
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
>>> of redundant sto
>>> rage directories!
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>> dataloss due to lack of re
>>> dundant storage directories!
>>>
>>>
>>> ************************************************************/
>>> 2013-02-28 05:06:23,385 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
>>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
>>> 2013-02-28 05:06:23,556 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
>>> at 10 second(s).
>>> 2013-02-28 05:06:23,556 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>>> started
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
>>> of redundant sto
>>> rage directories!
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>> dataloss due to lack of re
>>> dundant storage directories!
>>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
>>> Refreshing hosts (include/exclude) list
>>> 2013-02-28 05:06:25,623 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>> dfs.block.invalidate.limit=1000
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> dfs.block.access.token.enable=false
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> defaultReplication = 1
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
>>> = 512
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
>>> = 1
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> maxReplicationStreams = 2
>>> 2013-02-28 05:06:26,016 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> shouldCheckForEnoughRacks = false
>>> 2013-02-28 05:06:26,016 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> replicationRecheckInterval = 3000
>>> 2013-02-28 05:06:26,016 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> encryptDataTransfer = false
>>> 2013-02-28 05:06:26,022 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
>>> hdfs (auth:SIMPLE)
>>> 2013-02-28 05:06:26,022 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
>>> hadmin
>>> 2013-02-28 05:06:26,022 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
>>> true
>>> 2013-02-28 05:06:26,023 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>> 2013-02-28 05:06:26,026 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>>> 2013-02-28 05:06:26,359 INFO
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring
>>> more than 10 times
>>> 2013-02-28 05:06:26,361 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>> 2013-02-28 05:06:26,361 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> dfs.namenode.safemode.min.datanodes = 0
>>> 2013-02-28 05:06:26,361 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> dfs.namenode.safemode.extension = 0
>>> 2013-02-28 05:06:26,378 INFO
>>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>>> 7275@OPERA-MAST1.ny.os.local
>>> 2013-02-28 05:06:26,381 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
>>> system...
>>> 2013-02-28 05:06:26,381 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>>> stopped.
>>> 2013-02-28 05:06:26,381 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>>> shutdown complete.
>>> 2013-02-28 05:06:26,382 FATAL
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>>> java.io.IOException: NameNode is not formatted.
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>> with status 1
>>> 2013-02-28 05:06:26,394 INFO
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>>>
>>>
>>>
>>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
>>> <pr...@gmail.com> wrote:
>>>>
>>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
>>>> If I do compare with the older path the permissions are same on the parent
>>>> directories.
>>>> Do I need to create this this directory manually and set the permission
>>>> ?
>>>>
>>>> Older Path
>>>>
>>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>>>> total 4
>>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>>>>
>>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>>>> total 12
>>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>>
>>>>
>>>> New Path
>>>>
>>>> # ll /mnt/san1/hdfs/cache/hdfs/
>>>> total 4
>>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>>>>
>>>>
>>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>>>> total 4
>>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>
>>>>
>>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> The exact error is displayed on your log and should be somewhat self
>>>>> explanatory:
>>>>>
>>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>>>>> state: storage directory does not exist or is not accessible.
>>>>>
>>>>> Please check this one's availability, permissions (the NN user should
>>>>> be able to access it).
>>>>>
>>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>>>>> <pr...@gmail.com> wrote:
>>>>> > Please find below logs for shutting down the namenode service. Can
>>>>> > anybody
>>>>> > check this
>>>>> >
>>>>> > 2013-02-28 02:07:51,752 WARN
>>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>>> > configuration files. Please update hdfs configuration.
>>>>> > 2013-02-28 02:07:51,754 WARN
>>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>>> > configuration files. Please update hdfs configuration.
>>>>> > 2013-02-28 02:07:51,754 WARN
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>>>>> > storage
>>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>>>>> > to lack
>>>>> > of redundant storage directories!
>>>>> > 2013-02-28 02:07:51,754 WARN
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>>>>> > namespace
>>>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware
>>>>> > of
>>>>> > dataloss due to lack of redundant storage directories!
>>>>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>>>>> > Refreshing hosts (include/exclude) list
>>>>> > 2013-02-28 02:07:51,890 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>>>> > dfs.block.invalidate.limit=1000
>>>>> > 2013-02-28 02:07:51,909 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > dfs.block.access.token.enable=false
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > defaultReplication = 1
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > maxReplication
>>>>> > = 512
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > minReplication
>>>>> > = 1
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > maxReplicationStreams = 2
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > shouldCheckForEnoughRacks = false
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > replicationRecheckInterval = 3000
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > encryptDataTransfer = false
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>>>>> > =
>>>>> > hdfs (auth:SIMPLE)
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>>>>> > =
>>>>> > hadmin
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > isPermissionEnabled =
>>>>> > true
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
>>>>> > false
>>>>> > 2013-02-28 02:07:51,925 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>>>>> > true
>>>>> > 2013-02-28 02:07:52,462 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>>>>> > occuring
>>>>> > more than 10 times
>>>>> > 2013-02-28 02:07:52,466 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>>>> > 2013-02-28 02:07:52,467 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > dfs.namenode.safemode.min.datanodes = 0
>>>>> > 2013-02-28 02:07:52,467 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > dfs.namenode.safemode.extension = 0
>>>>> > 2013-02-28 02:07:52,469 INFO
>>>>> > org.apache.hadoop.hdfs.server.common.Storage:
>>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>>>>> > 2013-02-28 02:07:52,471 INFO
>>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>>>>> > metrics
>>>>> > system...
>>>>> > 2013-02-28 02:07:52,472 INFO
>>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>>> > system
>>>>> > stopped.
>>>>> > 2013-02-28 02:07:52,473 INFO
>>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>>> > system
>>>>> > shutdown complete.
>>>>> > 2013-02-28 02:07:52,473 FATAL
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>>>>> > namenode join
>>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>>> > Directory
>>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
>>>>> > storage
>>>>> > directory does not exist or is not accessible.
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>>>> > with
>>>>> > status 1
>>>>> > 2013-02-28 02:08:48,913 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>>>> > /************************************************************
>>>>> > SHUTDOWN_MSG: Shutting down NameNode at
>>>>> > OPERA-MAST1.ny.os.local/192.168.1.3
>>>>> >
>>>>> >
>>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>>>>> > <pr...@gmail.com> wrote:
>>>>> >>
>>>>> >> Hi Guys,
>>>>> >>
>>>>> >> I have space on other partition. Can I change the path for cache
>>>>> >> files on
>>>>> >> other partition ? I have below properties . Can it resolve the issue
>>>>> >> ? If i
>>>>> >> change the path to other directories and restart services I get the
>>>>> >> below
>>>>> >> error while starting the service namenode. I didn't find anything in
>>>>> >> logs so
>>>>> >> far. Can you please suggest something ?
>>>>> >>
>>>>> >> <property>
>>>>> >> <name>hadoop.tmp.dir</name>
>>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>>>>> >> </property>
>>>>> >> <property>
>>>>> >> <name>dfs.namenode.name.dir</name>
>>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>>>>> >> </property>
>>>>> >> <property>
>>>>> >> <name>dfs.namenode.checkpoint.dir</name>
>>>>> >>
>>>>> >>
>>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/namesecondary</value>
>>>>> >> </property>
>>>>> >> <property>
>>>>> >>
>>>>> >>
>>>>> >> Service namenode is failing
>>>>> >>
>>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
>>>>> >> status; done
>>>>> >> Hadoop datanode is running [ OK ]
>>>>> >> Hadoop namenode is dead and pid file exists [FAILED]
>>>>> >> Hadoop secondarynamenode is running [ OK ]
>>>>> >>
>>>>> >> Thanks,
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>>>>> >> <pr...@gmail.com> wrote:
>>>>> >>>
>>>>> >>>
>>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>>>>> >>> wrote:
>>>>> >>>>
>>>>> >>>> http://NNHOST:50070/conf
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> Harsh, I changed the value as said & restarted service NN. For
>>>>> >>> verifying
>>>>> >>> i checked the http link that you gave and i saw the property their
>>>>> >>> but on
>>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>>>>> >>> missing
>>>>> >>> blocks. Please check the logs or run fsck in order to identify the
>>>>> >>> missing
>>>>> >>> blocks.) when i clicked on this link i can see file names . Do I
>>>>> >>> need to
>>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>>>>> >>> command fsck
>>>>> >>> that i can run on the running hadoop ?
>>>>> >>>
>>>>> >>> Thanks,
>>>>> >>>
>>>>> >>
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>
>>>>
>>>
>>
>
--
Harsh J
Re: NameNode low on available disk space
Posted by Harsh J <ha...@cloudera.com>.
I believe I already responded to another one of your multiple threads
on this problem, that you have to move contents of the old dir to the
new dir and not format. Thats how one changes the dfs.name.dir
properly.
On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
<pr...@gmail.com> wrote:
> Hi,
>
> May i know the answer plz ?
>
> Thanks,
>
>
> On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
> <pr...@gmail.com> wrote:
>>
>> Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>> and set permission . when i restart hadoop services. It removes and I find
>> below logs.
>>
>> Do I need to format the NN?
>> Below is the command to format the NN ?
>> Any kind of loss while formatting ?
>> Is there any way to avoid formatting and change the cache path ?
>>
>> 2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
>> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> 81133@OPERA-MAST1.ny.os.local
>> 2013-02-28 05:57:50,904 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
>> system...
>> 2013-02-28 05:57:50,904 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> stopped.
>> 2013-02-28 05:57:50,904 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> shutdown complete.
>> 2013-02-28 05:57:50,905 FATAL
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>> java.io.IOException: NameNode is not formatted.
>>
>> Command to format the NN.
>>
>> sudo -u hdfs hdfs namenode -format
>>
>> Thanks,
>>
>>
>> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
>> <pr...@gmail.com> wrote:
>>>
>>> After creating the directory and setting permission I tried to restart
>>> the services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>>> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are not
>>> being started.
>>>
>>> Need to check few logs from below logs.
>>> ===================================
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
>>> of redundant sto
>>> rage directories!
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>> dataloss due to lack of re
>>> dundant storage directories!
>>>
>>>
>>> ************************************************************/
>>> 2013-02-28 05:06:23,385 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
>>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
>>> 2013-02-28 05:06:23,556 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
>>> at 10 second(s).
>>> 2013-02-28 05:06:23,556 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>>> started
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
>>> of redundant sto
>>> rage directories!
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>> dataloss due to lack of re
>>> dundant storage directories!
>>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
>>> Refreshing hosts (include/exclude) list
>>> 2013-02-28 05:06:25,623 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>> dfs.block.invalidate.limit=1000
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> dfs.block.access.token.enable=false
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> defaultReplication = 1
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
>>> = 512
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
>>> = 1
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> maxReplicationStreams = 2
>>> 2013-02-28 05:06:26,016 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> shouldCheckForEnoughRacks = false
>>> 2013-02-28 05:06:26,016 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> replicationRecheckInterval = 3000
>>> 2013-02-28 05:06:26,016 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> encryptDataTransfer = false
>>> 2013-02-28 05:06:26,022 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
>>> hdfs (auth:SIMPLE)
>>> 2013-02-28 05:06:26,022 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
>>> hadmin
>>> 2013-02-28 05:06:26,022 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
>>> true
>>> 2013-02-28 05:06:26,023 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>> 2013-02-28 05:06:26,026 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>>> 2013-02-28 05:06:26,359 INFO
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring
>>> more than 10 times
>>> 2013-02-28 05:06:26,361 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>> 2013-02-28 05:06:26,361 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> dfs.namenode.safemode.min.datanodes = 0
>>> 2013-02-28 05:06:26,361 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> dfs.namenode.safemode.extension = 0
>>> 2013-02-28 05:06:26,378 INFO
>>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>>> 7275@OPERA-MAST1.ny.os.local
>>> 2013-02-28 05:06:26,381 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
>>> system...
>>> 2013-02-28 05:06:26,381 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>>> stopped.
>>> 2013-02-28 05:06:26,381 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>>> shutdown complete.
>>> 2013-02-28 05:06:26,382 FATAL
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>>> java.io.IOException: NameNode is not formatted.
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>> with status 1
>>> 2013-02-28 05:06:26,394 INFO
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>>>
>>>
>>>
>>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
>>> <pr...@gmail.com> wrote:
>>>>
>>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
>>>> If I do compare with the older path the permissions are same on the parent
>>>> directories.
>>>> Do I need to create this this directory manually and set the permission
>>>> ?
>>>>
>>>> Older Path
>>>>
>>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>>>> total 4
>>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>>>>
>>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>>>> total 12
>>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>>
>>>>
>>>> New Path
>>>>
>>>> # ll /mnt/san1/hdfs/cache/hdfs/
>>>> total 4
>>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>>>>
>>>>
>>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>>>> total 4
>>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>
>>>>
>>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> The exact error is displayed on your log and should be somewhat self
>>>>> explanatory:
>>>>>
>>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>>>>> state: storage directory does not exist or is not accessible.
>>>>>
>>>>> Please check this one's availability, permissions (the NN user should
>>>>> be able to access it).
>>>>>
>>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>>>>> <pr...@gmail.com> wrote:
>>>>> > Please find below logs for shutting down the namenode service. Can
>>>>> > anybody
>>>>> > check this
>>>>> >
>>>>> > 2013-02-28 02:07:51,752 WARN
>>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>>> > configuration files. Please update hdfs configuration.
>>>>> > 2013-02-28 02:07:51,754 WARN
>>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>>> > configuration files. Please update hdfs configuration.
>>>>> > 2013-02-28 02:07:51,754 WARN
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>>>>> > storage
>>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>>>>> > to lack
>>>>> > of redundant storage directories!
>>>>> > 2013-02-28 02:07:51,754 WARN
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>>>>> > namespace
>>>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware
>>>>> > of
>>>>> > dataloss due to lack of redundant storage directories!
>>>>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>>>>> > Refreshing hosts (include/exclude) list
>>>>> > 2013-02-28 02:07:51,890 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>>>> > dfs.block.invalidate.limit=1000
>>>>> > 2013-02-28 02:07:51,909 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > dfs.block.access.token.enable=false
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > defaultReplication = 1
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > maxReplication
>>>>> > = 512
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > minReplication
>>>>> > = 1
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > maxReplicationStreams = 2
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > shouldCheckForEnoughRacks = false
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > replicationRecheckInterval = 3000
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > encryptDataTransfer = false
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>>>>> > =
>>>>> > hdfs (auth:SIMPLE)
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>>>>> > =
>>>>> > hadmin
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > isPermissionEnabled =
>>>>> > true
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
>>>>> > false
>>>>> > 2013-02-28 02:07:51,925 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>>>>> > true
>>>>> > 2013-02-28 02:07:52,462 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>>>>> > occuring
>>>>> > more than 10 times
>>>>> > 2013-02-28 02:07:52,466 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>>>> > 2013-02-28 02:07:52,467 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > dfs.namenode.safemode.min.datanodes = 0
>>>>> > 2013-02-28 02:07:52,467 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > dfs.namenode.safemode.extension = 0
>>>>> > 2013-02-28 02:07:52,469 INFO
>>>>> > org.apache.hadoop.hdfs.server.common.Storage:
>>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>>>>> > 2013-02-28 02:07:52,471 INFO
>>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>>>>> > metrics
>>>>> > system...
>>>>> > 2013-02-28 02:07:52,472 INFO
>>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>>> > system
>>>>> > stopped.
>>>>> > 2013-02-28 02:07:52,473 INFO
>>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>>> > system
>>>>> > shutdown complete.
>>>>> > 2013-02-28 02:07:52,473 FATAL
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>>>>> > namenode join
>>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>>> > Directory
>>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
>>>>> > storage
>>>>> > directory does not exist or is not accessible.
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>>>> > with
>>>>> > status 1
>>>>> > 2013-02-28 02:08:48,913 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>>>> > /************************************************************
>>>>> > SHUTDOWN_MSG: Shutting down NameNode at
>>>>> > OPERA-MAST1.ny.os.local/192.168.1.3
>>>>> >
>>>>> >
>>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>>>>> > <pr...@gmail.com> wrote:
>>>>> >>
>>>>> >> Hi Guys,
>>>>> >>
>>>>> >> I have space on other partition. Can I change the path for cache
>>>>> >> files on
>>>>> >> other partition ? I have below properties . Can it resolve the issue
>>>>> >> ? If i
>>>>> >> change the path to other directories and restart services I get the
>>>>> >> below
>>>>> >> error while starting the service namenode. I didn't find anything in
>>>>> >> logs so
>>>>> >> far. Can you please suggest something ?
>>>>> >>
>>>>> >> <property>
>>>>> >> <name>hadoop.tmp.dir</name>
>>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>>>>> >> </property>
>>>>> >> <property>
>>>>> >> <name>dfs.namenode.name.dir</name>
>>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>>>>> >> </property>
>>>>> >> <property>
>>>>> >> <name>dfs.namenode.checkpoint.dir</name>
>>>>> >>
>>>>> >>
>>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/namesecondary</value>
>>>>> >> </property>
>>>>> >> <property>
>>>>> >>
>>>>> >>
>>>>> >> Service namenode is failing
>>>>> >>
>>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
>>>>> >> status; done
>>>>> >> Hadoop datanode is running [ OK ]
>>>>> >> Hadoop namenode is dead and pid file exists [FAILED]
>>>>> >> Hadoop secondarynamenode is running [ OK ]
>>>>> >>
>>>>> >> Thanks,
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>>>>> >> <pr...@gmail.com> wrote:
>>>>> >>>
>>>>> >>>
>>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>>>>> >>> wrote:
>>>>> >>>>
>>>>> >>>> http://NNHOST:50070/conf
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> Harsh, I changed the value as said & restarted service NN. For
>>>>> >>> verifying
>>>>> >>> i checked the http link that you gave and i saw the property their
>>>>> >>> but on
>>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>>>>> >>> missing
>>>>> >>> blocks. Please check the logs or run fsck in order to identify the
>>>>> >>> missing
>>>>> >>> blocks.) when i clicked on this link i can see file names . Do I
>>>>> >>> need to
>>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>>>>> >>> command fsck
>>>>> >>> that i can run on the running hadoop ?
>>>>> >>>
>>>>> >>> Thanks,
>>>>> >>>
>>>>> >>
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>
>>>>
>>>
>>
>
--
Harsh J
Re: NameNode low on available disk space
Posted by Harsh J <ha...@cloudera.com>.
I believe I already responded to another one of your multiple threads
on this problem, that you have to move contents of the old dir to the
new dir and not format. Thats how one changes the dfs.name.dir
properly.
On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
<pr...@gmail.com> wrote:
> Hi,
>
> May i know the answer plz ?
>
> Thanks,
>
>
> On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
> <pr...@gmail.com> wrote:
>>
>> Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>> and set permission . when i restart hadoop services. It removes and I find
>> below logs.
>>
>> Do I need to format the NN?
>> Below is the command to format the NN ?
>> Any kind of loss while formatting ?
>> Is there any way to avoid formatting and change the cache path ?
>>
>> 2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
>> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> 81133@OPERA-MAST1.ny.os.local
>> 2013-02-28 05:57:50,904 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
>> system...
>> 2013-02-28 05:57:50,904 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> stopped.
>> 2013-02-28 05:57:50,904 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> shutdown complete.
>> 2013-02-28 05:57:50,905 FATAL
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>> java.io.IOException: NameNode is not formatted.
>>
>> Command to format the NN.
>>
>> sudo -u hdfs hdfs namenode -format
>>
>> Thanks,
>>
>>
>> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
>> <pr...@gmail.com> wrote:
>>>
>>> After creating the directory and setting permission I tried to restart
>>> the services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>>> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are not
>>> being started.
>>>
>>> Need to check few logs from below logs.
>>> ===================================
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
>>> of redundant sto
>>> rage directories!
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>> dataloss due to lack of re
>>> dundant storage directories!
>>>
>>>
>>> ************************************************************/
>>> 2013-02-28 05:06:23,385 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
>>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
>>> 2013-02-28 05:06:23,556 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
>>> at 10 second(s).
>>> 2013-02-28 05:06:23,556 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>>> started
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
>>> of redundant sto
>>> rage directories!
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>> dataloss due to lack of re
>>> dundant storage directories!
>>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
>>> Refreshing hosts (include/exclude) list
>>> 2013-02-28 05:06:25,623 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>> dfs.block.invalidate.limit=1000
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> dfs.block.access.token.enable=false
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> defaultReplication = 1
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
>>> = 512
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
>>> = 1
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> maxReplicationStreams = 2
>>> 2013-02-28 05:06:26,016 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> shouldCheckForEnoughRacks = false
>>> 2013-02-28 05:06:26,016 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> replicationRecheckInterval = 3000
>>> 2013-02-28 05:06:26,016 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> encryptDataTransfer = false
>>> 2013-02-28 05:06:26,022 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
>>> hdfs (auth:SIMPLE)
>>> 2013-02-28 05:06:26,022 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
>>> hadmin
>>> 2013-02-28 05:06:26,022 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
>>> true
>>> 2013-02-28 05:06:26,023 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>> 2013-02-28 05:06:26,026 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>>> 2013-02-28 05:06:26,359 INFO
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring
>>> more than 10 times
>>> 2013-02-28 05:06:26,361 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>> 2013-02-28 05:06:26,361 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> dfs.namenode.safemode.min.datanodes = 0
>>> 2013-02-28 05:06:26,361 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> dfs.namenode.safemode.extension = 0
>>> 2013-02-28 05:06:26,378 INFO
>>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>>> 7275@OPERA-MAST1.ny.os.local
>>> 2013-02-28 05:06:26,381 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
>>> system...
>>> 2013-02-28 05:06:26,381 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>>> stopped.
>>> 2013-02-28 05:06:26,381 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>>> shutdown complete.
>>> 2013-02-28 05:06:26,382 FATAL
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>>> java.io.IOException: NameNode is not formatted.
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>> with status 1
>>> 2013-02-28 05:06:26,394 INFO
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>>>
>>>
>>>
>>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
>>> <pr...@gmail.com> wrote:
>>>>
>>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
>>>> If I do compare with the older path the permissions are same on the parent
>>>> directories.
>>>> Do I need to create this this directory manually and set the permission
>>>> ?
>>>>
>>>> Older Path
>>>>
>>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>>>> total 4
>>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>>>>
>>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>>>> total 12
>>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>>
>>>>
>>>> New Path
>>>>
>>>> # ll /mnt/san1/hdfs/cache/hdfs/
>>>> total 4
>>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>>>>
>>>>
>>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>>>> total 4
>>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>
>>>>
>>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> The exact error is displayed on your log and should be somewhat self
>>>>> explanatory:
>>>>>
>>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>>>>> state: storage directory does not exist or is not accessible.
>>>>>
>>>>> Please check this one's availability, permissions (the NN user should
>>>>> be able to access it).
>>>>>
>>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>>>>> <pr...@gmail.com> wrote:
>>>>> > Please find below logs for shutting down the namenode service. Can
>>>>> > anybody
>>>>> > check this
>>>>> >
>>>>> > 2013-02-28 02:07:51,752 WARN
>>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>>> > configuration files. Please update hdfs configuration.
>>>>> > 2013-02-28 02:07:51,754 WARN
>>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>>> > configuration files. Please update hdfs configuration.
>>>>> > 2013-02-28 02:07:51,754 WARN
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>>>>> > storage
>>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>>>>> > to lack
>>>>> > of redundant storage directories!
>>>>> > 2013-02-28 02:07:51,754 WARN
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>>>>> > namespace
>>>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware
>>>>> > of
>>>>> > dataloss due to lack of redundant storage directories!
>>>>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>>>>> > Refreshing hosts (include/exclude) list
>>>>> > 2013-02-28 02:07:51,890 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>>>> > dfs.block.invalidate.limit=1000
>>>>> > 2013-02-28 02:07:51,909 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > dfs.block.access.token.enable=false
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > defaultReplication = 1
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > maxReplication
>>>>> > = 512
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > minReplication
>>>>> > = 1
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > maxReplicationStreams = 2
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > shouldCheckForEnoughRacks = false
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > replicationRecheckInterval = 3000
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > encryptDataTransfer = false
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>>>>> > =
>>>>> > hdfs (auth:SIMPLE)
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>>>>> > =
>>>>> > hadmin
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > isPermissionEnabled =
>>>>> > true
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
>>>>> > false
>>>>> > 2013-02-28 02:07:51,925 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>>>>> > true
>>>>> > 2013-02-28 02:07:52,462 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>>>>> > occuring
>>>>> > more than 10 times
>>>>> > 2013-02-28 02:07:52,466 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>>>> > 2013-02-28 02:07:52,467 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > dfs.namenode.safemode.min.datanodes = 0
>>>>> > 2013-02-28 02:07:52,467 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > dfs.namenode.safemode.extension = 0
>>>>> > 2013-02-28 02:07:52,469 INFO
>>>>> > org.apache.hadoop.hdfs.server.common.Storage:
>>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>>>>> > 2013-02-28 02:07:52,471 INFO
>>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>>>>> > metrics
>>>>> > system...
>>>>> > 2013-02-28 02:07:52,472 INFO
>>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>>> > system
>>>>> > stopped.
>>>>> > 2013-02-28 02:07:52,473 INFO
>>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>>> > system
>>>>> > shutdown complete.
>>>>> > 2013-02-28 02:07:52,473 FATAL
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>>>>> > namenode join
>>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>>> > Directory
>>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
>>>>> > storage
>>>>> > directory does not exist or is not accessible.
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>>>> > with
>>>>> > status 1
>>>>> > 2013-02-28 02:08:48,913 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>>>> > /************************************************************
>>>>> > SHUTDOWN_MSG: Shutting down NameNode at
>>>>> > OPERA-MAST1.ny.os.local/192.168.1.3
>>>>> >
>>>>> >
>>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>>>>> > <pr...@gmail.com> wrote:
>>>>> >>
>>>>> >> Hi Guys,
>>>>> >>
>>>>> >> I have space on other partition. Can I change the path for cache
>>>>> >> files on
>>>>> >> other partition ? I have below properties . Can it resolve the issue
>>>>> >> ? If i
>>>>> >> change the path to other directories and restart services I get the
>>>>> >> below
>>>>> >> error while starting the service namenode. I didn't find anything in
>>>>> >> logs so
>>>>> >> far. Can you please suggest something ?
>>>>> >>
>>>>> >> <property>
>>>>> >> <name>hadoop.tmp.dir</name>
>>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>>>>> >> </property>
>>>>> >> <property>
>>>>> >> <name>dfs.namenode.name.dir</name>
>>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>>>>> >> </property>
>>>>> >> <property>
>>>>> >> <name>dfs.namenode.checkpoint.dir</name>
>>>>> >>
>>>>> >>
>>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/namesecondary</value>
>>>>> >> </property>
>>>>> >> <property>
>>>>> >>
>>>>> >>
>>>>> >> Service namenode is failing
>>>>> >>
>>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
>>>>> >> status; done
>>>>> >> Hadoop datanode is running [ OK ]
>>>>> >> Hadoop namenode is dead and pid file exists [FAILED]
>>>>> >> Hadoop secondarynamenode is running [ OK ]
>>>>> >>
>>>>> >> Thanks,
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>>>>> >> <pr...@gmail.com> wrote:
>>>>> >>>
>>>>> >>>
>>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>>>>> >>> wrote:
>>>>> >>>>
>>>>> >>>> http://NNHOST:50070/conf
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> Harsh, I changed the value as said & restarted service NN. For
>>>>> >>> verifying
>>>>> >>> i checked the http link that you gave and i saw the property their
>>>>> >>> but on
>>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>>>>> >>> missing
>>>>> >>> blocks. Please check the logs or run fsck in order to identify the
>>>>> >>> missing
>>>>> >>> blocks.) when i clicked on this link i can see file names . Do I
>>>>> >>> need to
>>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>>>>> >>> command fsck
>>>>> >>> that i can run on the running hadoop ?
>>>>> >>>
>>>>> >>> Thanks,
>>>>> >>>
>>>>> >>
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>
>>>>
>>>
>>
>
--
Harsh J
Re: NameNode low on available disk space
Posted by Harsh J <ha...@cloudera.com>.
I believe I already responded to another one of your multiple threads
on this problem, that you have to move contents of the old dir to the
new dir and not format. Thats how one changes the dfs.name.dir
properly.
On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
<pr...@gmail.com> wrote:
> Hi,
>
> May i know the answer plz ?
>
> Thanks,
>
>
> On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
> <pr...@gmail.com> wrote:
>>
>> Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>> and set permission . when i restart hadoop services. It removes and I find
>> below logs.
>>
>> Do I need to format the NN?
>> Below is the command to format the NN ?
>> Any kind of loss while formatting ?
>> Is there any way to avoid formatting and change the cache path ?
>>
>> 2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
>> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> 81133@OPERA-MAST1.ny.os.local
>> 2013-02-28 05:57:50,904 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
>> system...
>> 2013-02-28 05:57:50,904 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> stopped.
>> 2013-02-28 05:57:50,904 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> shutdown complete.
>> 2013-02-28 05:57:50,905 FATAL
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>> java.io.IOException: NameNode is not formatted.
>>
>> Command to format the NN.
>>
>> sudo -u hdfs hdfs namenode -format
>>
>> Thanks,
>>
>>
>> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
>> <pr...@gmail.com> wrote:
>>>
>>> After creating the directory and setting permission I tried to restart
>>> the services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
>>> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are not
>>> being started.
>>>
>>> Need to check few logs from below logs.
>>> ===================================
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
>>> of redundant sto
>>> rage directories!
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>> dataloss due to lack of re
>>> dundant storage directories!
>>>
>>>
>>> ************************************************************/
>>> 2013-02-28 05:06:23,385 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
>>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
>>> 2013-02-28 05:06:23,556 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
>>> at 10 second(s).
>>> 2013-02-28 05:06:23,556 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>>> started
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> configuration files. Please update hdfs configuration.
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
>>> of redundant sto
>>> rage directories!
>>> 2013-02-28 05:06:24,906 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>> dataloss due to lack of re
>>> dundant storage directories!
>>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
>>> Refreshing hosts (include/exclude) list
>>> 2013-02-28 05:06:25,623 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>> dfs.block.invalidate.limit=1000
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> dfs.block.access.token.enable=false
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> defaultReplication = 1
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
>>> = 512
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
>>> = 1
>>> 2013-02-28 05:06:26,015 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> maxReplicationStreams = 2
>>> 2013-02-28 05:06:26,016 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> shouldCheckForEnoughRacks = false
>>> 2013-02-28 05:06:26,016 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> replicationRecheckInterval = 3000
>>> 2013-02-28 05:06:26,016 INFO
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> encryptDataTransfer = false
>>> 2013-02-28 05:06:26,022 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
>>> hdfs (auth:SIMPLE)
>>> 2013-02-28 05:06:26,022 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
>>> hadmin
>>> 2013-02-28 05:06:26,022 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
>>> true
>>> 2013-02-28 05:06:26,023 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>> 2013-02-28 05:06:26,026 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>>> 2013-02-28 05:06:26,359 INFO
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring
>>> more than 10 times
>>> 2013-02-28 05:06:26,361 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>> 2013-02-28 05:06:26,361 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> dfs.namenode.safemode.min.datanodes = 0
>>> 2013-02-28 05:06:26,361 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> dfs.namenode.safemode.extension = 0
>>> 2013-02-28 05:06:26,378 INFO
>>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>>> 7275@OPERA-MAST1.ny.os.local
>>> 2013-02-28 05:06:26,381 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
>>> system...
>>> 2013-02-28 05:06:26,381 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>>> stopped.
>>> 2013-02-28 05:06:26,381 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>>> shutdown complete.
>>> 2013-02-28 05:06:26,382 FATAL
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>>> java.io.IOException: NameNode is not formatted.
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>> with status 1
>>> 2013-02-28 05:06:26,394 INFO
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>>>
>>>
>>>
>>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
>>> <pr...@gmail.com> wrote:
>>>>
>>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
>>>> If I do compare with the older path the permissions are same on the parent
>>>> directories.
>>>> Do I need to create this this directory manually and set the permission
>>>> ?
>>>>
>>>> Older Path
>>>>
>>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>>>> total 4
>>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>>>>
>>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>>>> total 12
>>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>>
>>>>
>>>> New Path
>>>>
>>>> # ll /mnt/san1/hdfs/cache/hdfs/
>>>> total 4
>>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>>>>
>>>>
>>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>>>> total 4
>>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>
>>>>
>>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> The exact error is displayed on your log and should be somewhat self
>>>>> explanatory:
>>>>>
>>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>>>>> state: storage directory does not exist or is not accessible.
>>>>>
>>>>> Please check this one's availability, permissions (the NN user should
>>>>> be able to access it).
>>>>>
>>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>>>>> <pr...@gmail.com> wrote:
>>>>> > Please find below logs for shutting down the namenode service. Can
>>>>> > anybody
>>>>> > check this
>>>>> >
>>>>> > 2013-02-28 02:07:51,752 WARN
>>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>>> > configuration files. Please update hdfs configuration.
>>>>> > 2013-02-28 02:07:51,754 WARN
>>>>> > org.apache.hadoop.hdfs.server.common.Util: Path
>>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>>> > configuration files. Please update hdfs configuration.
>>>>> > 2013-02-28 02:07:51,754 WARN
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>>>>> > storage
>>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>>>>> > to lack
>>>>> > of redundant storage directories!
>>>>> > 2013-02-28 02:07:51,754 WARN
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>>>>> > namespace
>>>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware
>>>>> > of
>>>>> > dataloss due to lack of redundant storage directories!
>>>>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>>>>> > Refreshing hosts (include/exclude) list
>>>>> > 2013-02-28 02:07:51,890 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>>>> > dfs.block.invalidate.limit=1000
>>>>> > 2013-02-28 02:07:51,909 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > dfs.block.access.token.enable=false
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > defaultReplication = 1
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > maxReplication
>>>>> > = 512
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > minReplication
>>>>> > = 1
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > maxReplicationStreams = 2
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > shouldCheckForEnoughRacks = false
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > replicationRecheckInterval = 3000
>>>>> > 2013-02-28 02:07:51,910 INFO
>>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>>> > encryptDataTransfer = false
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>>>>> > =
>>>>> > hdfs (auth:SIMPLE)
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>>>>> > =
>>>>> > hadmin
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > isPermissionEnabled =
>>>>> > true
>>>>> > 2013-02-28 02:07:51,920 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
>>>>> > false
>>>>> > 2013-02-28 02:07:51,925 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>>>>> > true
>>>>> > 2013-02-28 02:07:52,462 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>>>>> > occuring
>>>>> > more than 10 times
>>>>> > 2013-02-28 02:07:52,466 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>>>> > 2013-02-28 02:07:52,467 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > dfs.namenode.safemode.min.datanodes = 0
>>>>> > 2013-02-28 02:07:52,467 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>> > dfs.namenode.safemode.extension = 0
>>>>> > 2013-02-28 02:07:52,469 INFO
>>>>> > org.apache.hadoop.hdfs.server.common.Storage:
>>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>>>>> > 2013-02-28 02:07:52,471 INFO
>>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>>>>> > metrics
>>>>> > system...
>>>>> > 2013-02-28 02:07:52,472 INFO
>>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>>> > system
>>>>> > stopped.
>>>>> > 2013-02-28 02:07:52,473 INFO
>>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>>> > system
>>>>> > shutdown complete.
>>>>> > 2013-02-28 02:07:52,473 FATAL
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>>>>> > namenode join
>>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>>> > Directory
>>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
>>>>> > storage
>>>>> > directory does not exist or is not accessible.
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>>>> > at
>>>>> >
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>>>> > with
>>>>> > status 1
>>>>> > 2013-02-28 02:08:48,913 INFO
>>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>>>> > /************************************************************
>>>>> > SHUTDOWN_MSG: Shutting down NameNode at
>>>>> > OPERA-MAST1.ny.os.local/192.168.1.3
>>>>> >
>>>>> >
>>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>>>>> > <pr...@gmail.com> wrote:
>>>>> >>
>>>>> >> Hi Guys,
>>>>> >>
>>>>> >> I have space on other partition. Can I change the path for cache
>>>>> >> files on
>>>>> >> other partition ? I have below properties . Can it resolve the issue
>>>>> >> ? If i
>>>>> >> change the path to other directories and restart services I get the
>>>>> >> below
>>>>> >> error while starting the service namenode. I didn't find anything in
>>>>> >> logs so
>>>>> >> far. Can you please suggest something ?
>>>>> >>
>>>>> >> <property>
>>>>> >> <name>hadoop.tmp.dir</name>
>>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>>>>> >> </property>
>>>>> >> <property>
>>>>> >> <name>dfs.namenode.name.dir</name>
>>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>>>>> >> </property>
>>>>> >> <property>
>>>>> >> <name>dfs.namenode.checkpoint.dir</name>
>>>>> >>
>>>>> >>
>>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/namesecondary</value>
>>>>> >> </property>
>>>>> >> <property>
>>>>> >>
>>>>> >>
>>>>> >> Service namenode is failing
>>>>> >>
>>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
>>>>> >> status; done
>>>>> >> Hadoop datanode is running [ OK ]
>>>>> >> Hadoop namenode is dead and pid file exists [FAILED]
>>>>> >> Hadoop secondarynamenode is running [ OK ]
>>>>> >>
>>>>> >> Thanks,
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>>>>> >> <pr...@gmail.com> wrote:
>>>>> >>>
>>>>> >>>
>>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>>>>> >>> wrote:
>>>>> >>>>
>>>>> >>>> http://NNHOST:50070/conf
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> Harsh, I changed the value as said & restarted service NN. For
>>>>> >>> verifying
>>>>> >>> i checked the http link that you gave and i saw the property their
>>>>> >>> but on
>>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>>>>> >>> missing
>>>>> >>> blocks. Please check the logs or run fsck in order to identify the
>>>>> >>> missing
>>>>> >>> blocks.) when i clicked on this link i can see file names . Do I
>>>>> >>> need to
>>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>>>>> >>> command fsck
>>>>> >>> that i can run on the running hadoop ?
>>>>> >>>
>>>>> >>> Thanks,
>>>>> >>>
>>>>> >>
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>
>>>>
>>>
>>
>
--
Harsh J
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
Hi,
May i know the answer plz ?
Thanks,
On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> Even i created the
> file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock and set permission .
> when i restart hadoop services. It removes and I find below logs.
>
> Do I need to format the NN?
> Below is the command to format the NN ?
> Any kind of loss while formatting ?
> Is there any way to avoid formatting and change the cache path ?
>
> 2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
> 81133@OPERA-MAST1.ny.os.local
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 05:57:50,905 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> java.io.IOException: NameNode is not formatted.
>
> Command to format the NN.
>
> sudo -u hdfs hdfs namenode -format
>
> Thanks,
>
>
> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> After creating the directory and setting permission I tried to restart
>> the services and i get error
>> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> 7275@OPERA-MAST1.ny.os.local" and services are not being started.
>>
>> Need to check few logs from below logs.
>> ===================================
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack of redundant sto
>> rage directories!
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> dataloss due to lack of re
>> dundant storage directories!
>>
>>
>> ************************************************************/
>> 2013-02-28 05:06:23,385 WARN
>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
>> 2013-02-28 05:06:23,556 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2013-02-28 05:06:23,556 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> started
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack of redundant sto
>> rage directories!
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> dataloss due to lack of re
>> dundant storage directories!
>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
>> Refreshing hosts (include/exclude) list
>> 2013-02-28 05:06:25,623 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> dfs.block.invalidate.limit=1000
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> dfs.block.access.token.enable=false
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> defaultReplication = 1
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
>> = 512
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
>> = 1
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplicationStreams = 2
>> 2013-02-28 05:06:26,016 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> shouldCheckForEnoughRacks = false
>> 2013-02-28 05:06:26,016 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> replicationRecheckInterval = 3000
>> 2013-02-28 05:06:26,016 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> encryptDataTransfer = false
>> 2013-02-28 05:06:26,022 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
>> hdfs (auth:SIMPLE)
>> 2013-02-28 05:06:26,022 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
>> hadmin
>> 2013-02-28 05:06:26,022 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
>> true
>> 2013-02-28 05:06:26,023 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>> 2013-02-28 05:06:26,026 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>> 2013-02-28 05:06:26,359 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring more than 10 times
>> 2013-02-28 05:06:26,361 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> 2013-02-28 05:06:26,361 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.min.datanodes = 0
>> 2013-02-28 05:06:26,361 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.extension = 0
>> 2013-02-28 05:06:26,378 INFO
>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> 7275@OPERA-MAST1.ny.os.local
>> 2013-02-28 05:06:26,381 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics system...
>> 2013-02-28 05:06:26,381 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> stopped.
>> 2013-02-28 05:06:26,381 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> shutdown complete.
>> 2013-02-28 05:06:26,382 FATAL
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>> java.io.IOException: NameNode is not formatted.
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> with status 1
>> 2013-02-28 05:06:26,394 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>>
>>
>>
>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera <
>> project.linux.proj@gmail.com> wrote:
>>
>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created
>>> . If I do compare with the older path the permissions are same on the
>>> parent directories.
>>> Do I need to create this this directory manually and set the permission ?
>>>
>>> Older Path
>>>
>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>>> total 4
>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>>>
>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>>> total 12
>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>
>>>
>>> New Path
>>>
>>> # ll /mnt/san1/hdfs/cache/hdfs/
>>> total 4
>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>>>
>>>
>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>>> total 4
>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>
>>>
>>> Thanks,
>>>
>>>
>>>
>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> The exact error is displayed on your log and should be somewhat self
>>>> explanatory:
>>>>
>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>>>> state: storage directory does not exist or is not accessible.
>>>>
>>>> Please check this one's availability, permissions (the NN user should
>>>> be able to access it).
>>>>
>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>>>> <pr...@gmail.com> wrote:
>>>> > Please find below logs for shutting down the namenode service. Can
>>>> anybody
>>>> > check this
>>>> >
>>>> > 2013-02-28 02:07:51,752 WARN
>>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>> > configuration files. Please update hdfs configuration.
>>>> > 2013-02-28 02:07:51,754 WARN
>>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>> > configuration files. Please update hdfs configuration.
>>>> > 2013-02-28 02:07:51,754 WARN
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>>>> storage
>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>>>> to lack
>>>> > of redundant storage directories!
>>>> > 2013-02-28 02:07:51,754 WARN
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>>>> namespace
>>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>>> > dataloss due to lack of redundant storage directories!
>>>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>>>> > Refreshing hosts (include/exclude) list
>>>> > 2013-02-28 02:07:51,890 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>>> > dfs.block.invalidate.limit=1000
>>>> > 2013-02-28 02:07:51,909 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > dfs.block.access.token.enable=false
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > defaultReplication = 1
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> maxReplication
>>>> > = 512
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> minReplication
>>>> > = 1
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > maxReplicationStreams = 2
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > shouldCheckForEnoughRacks = false
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > replicationRecheckInterval = 3000
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > encryptDataTransfer = false
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>>>> =
>>>> > hdfs (auth:SIMPLE)
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>>>> =
>>>> > hadmin
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> isPermissionEnabled =
>>>> > true
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>>> > 2013-02-28 02:07:51,925 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>>>> true
>>>> > 2013-02-28 02:07:52,462 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>>>> occuring
>>>> > more than 10 times
>>>> > 2013-02-28 02:07:52,466 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>>> > 2013-02-28 02:07:52,467 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> > dfs.namenode.safemode.min.datanodes = 0
>>>> > 2013-02-28 02:07:52,467 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> > dfs.namenode.safemode.extension = 0
>>>> > 2013-02-28 02:07:52,469 INFO
>>>> org.apache.hadoop.hdfs.server.common.Storage:
>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>>>> > 2013-02-28 02:07:52,471 INFO
>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>>>> metrics
>>>> > system...
>>>> > 2013-02-28 02:07:52,472 INFO
>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>> system
>>>> > stopped.
>>>> > 2013-02-28 02:07:52,473 INFO
>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>> system
>>>> > shutdown complete.
>>>> > 2013-02-28 02:07:52,473 FATAL
>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>>>> namenode join
>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>> Directory
>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
>>>> storage
>>>> > directory does not exist or is not accessible.
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>>> with
>>>> > status 1
>>>> > 2013-02-28 02:08:48,913 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>>> > /************************************************************
>>>> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
>>>> 192.168.1.3
>>>> >
>>>> >
>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>>>> > <pr...@gmail.com> wrote:
>>>> >>
>>>> >> Hi Guys,
>>>> >>
>>>> >> I have space on other partition. Can I change the path for cache
>>>> files on
>>>> >> other partition ? I have below properties . Can it resolve the issue
>>>> ? If i
>>>> >> change the path to other directories and restart services I get the
>>>> below
>>>> >> error while starting the service namenode. I didn't find anything in
>>>> logs so
>>>> >> far. Can you please suggest something ?
>>>> >>
>>>> >> <property>
>>>> >> <name>hadoop.tmp.dir</name>
>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>>>> >> </property>
>>>> >> <property>
>>>> >> <name>dfs.namenode.name.dir</name>
>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>>>> >> </property>
>>>> >> <property>
>>>> >> <name>dfs.namenode.checkpoint.dir</name>
>>>> >>
>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>>>> }/dfs/namesecondary</value>
>>>> >> </property>
>>>> >> <property>
>>>> >>
>>>> >>
>>>> >> Service namenode is failing
>>>> >>
>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
>>>> status; done
>>>> >> Hadoop datanode is running [ OK ]
>>>> >> Hadoop namenode is dead and pid file exists [FAILED]
>>>> >> Hadoop secondarynamenode is running [ OK ]
>>>> >>
>>>> >> Thanks,
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>>>> >> <pr...@gmail.com> wrote:
>>>> >>>
>>>> >>>
>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>>>> wrote:
>>>> >>>>
>>>> >>>> http://NNHOST:50070/conf
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> Harsh, I changed the value as said & restarted service NN. For
>>>> verifying
>>>> >>> i checked the http link that you gave and i saw the property their
>>>> but on
>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>>>> missing
>>>> >>> blocks. Please check the logs or run fsck in order to identify the
>>>> missing
>>>> >>> blocks.) when i clicked on this link i can see file names . Do I
>>>> need to
>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>>>> command fsck
>>>> >>> that i can run on the running hadoop ?
>>>> >>>
>>>> >>> Thanks,
>>>> >>>
>>>> >>
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
Hi,
May i know the answer plz ?
Thanks,
On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> Even i created the
> file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock and set permission .
> when i restart hadoop services. It removes and I find below logs.
>
> Do I need to format the NN?
> Below is the command to format the NN ?
> Any kind of loss while formatting ?
> Is there any way to avoid formatting and change the cache path ?
>
> 2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
> 81133@OPERA-MAST1.ny.os.local
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 05:57:50,905 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> java.io.IOException: NameNode is not formatted.
>
> Command to format the NN.
>
> sudo -u hdfs hdfs namenode -format
>
> Thanks,
>
>
> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> After creating the directory and setting permission I tried to restart
>> the services and i get error
>> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> 7275@OPERA-MAST1.ny.os.local" and services are not being started.
>>
>> Need to check few logs from below logs.
>> ===================================
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack of redundant sto
>> rage directories!
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> dataloss due to lack of re
>> dundant storage directories!
>>
>>
>> ************************************************************/
>> 2013-02-28 05:06:23,385 WARN
>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
>> 2013-02-28 05:06:23,556 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2013-02-28 05:06:23,556 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> started
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack of redundant sto
>> rage directories!
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> dataloss due to lack of re
>> dundant storage directories!
>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
>> Refreshing hosts (include/exclude) list
>> 2013-02-28 05:06:25,623 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> dfs.block.invalidate.limit=1000
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> dfs.block.access.token.enable=false
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> defaultReplication = 1
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
>> = 512
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
>> = 1
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplicationStreams = 2
>> 2013-02-28 05:06:26,016 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> shouldCheckForEnoughRacks = false
>> 2013-02-28 05:06:26,016 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> replicationRecheckInterval = 3000
>> 2013-02-28 05:06:26,016 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> encryptDataTransfer = false
>> 2013-02-28 05:06:26,022 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
>> hdfs (auth:SIMPLE)
>> 2013-02-28 05:06:26,022 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
>> hadmin
>> 2013-02-28 05:06:26,022 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
>> true
>> 2013-02-28 05:06:26,023 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>> 2013-02-28 05:06:26,026 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>> 2013-02-28 05:06:26,359 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring more than 10 times
>> 2013-02-28 05:06:26,361 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> 2013-02-28 05:06:26,361 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.min.datanodes = 0
>> 2013-02-28 05:06:26,361 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.extension = 0
>> 2013-02-28 05:06:26,378 INFO
>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> 7275@OPERA-MAST1.ny.os.local
>> 2013-02-28 05:06:26,381 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics system...
>> 2013-02-28 05:06:26,381 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> stopped.
>> 2013-02-28 05:06:26,381 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> shutdown complete.
>> 2013-02-28 05:06:26,382 FATAL
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>> java.io.IOException: NameNode is not formatted.
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> with status 1
>> 2013-02-28 05:06:26,394 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>>
>>
>>
>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera <
>> project.linux.proj@gmail.com> wrote:
>>
>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created
>>> . If I do compare with the older path the permissions are same on the
>>> parent directories.
>>> Do I need to create this this directory manually and set the permission ?
>>>
>>> Older Path
>>>
>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>>> total 4
>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>>>
>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>>> total 12
>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>
>>>
>>> New Path
>>>
>>> # ll /mnt/san1/hdfs/cache/hdfs/
>>> total 4
>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>>>
>>>
>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>>> total 4
>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>
>>>
>>> Thanks,
>>>
>>>
>>>
>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> The exact error is displayed on your log and should be somewhat self
>>>> explanatory:
>>>>
>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>>>> state: storage directory does not exist or is not accessible.
>>>>
>>>> Please check this one's availability, permissions (the NN user should
>>>> be able to access it).
>>>>
>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>>>> <pr...@gmail.com> wrote:
>>>> > Please find below logs for shutting down the namenode service. Can
>>>> anybody
>>>> > check this
>>>> >
>>>> > 2013-02-28 02:07:51,752 WARN
>>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>> > configuration files. Please update hdfs configuration.
>>>> > 2013-02-28 02:07:51,754 WARN
>>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>> > configuration files. Please update hdfs configuration.
>>>> > 2013-02-28 02:07:51,754 WARN
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>>>> storage
>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>>>> to lack
>>>> > of redundant storage directories!
>>>> > 2013-02-28 02:07:51,754 WARN
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>>>> namespace
>>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>>> > dataloss due to lack of redundant storage directories!
>>>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>>>> > Refreshing hosts (include/exclude) list
>>>> > 2013-02-28 02:07:51,890 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>>> > dfs.block.invalidate.limit=1000
>>>> > 2013-02-28 02:07:51,909 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > dfs.block.access.token.enable=false
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > defaultReplication = 1
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> maxReplication
>>>> > = 512
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> minReplication
>>>> > = 1
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > maxReplicationStreams = 2
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > shouldCheckForEnoughRacks = false
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > replicationRecheckInterval = 3000
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > encryptDataTransfer = false
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>>>> =
>>>> > hdfs (auth:SIMPLE)
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>>>> =
>>>> > hadmin
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> isPermissionEnabled =
>>>> > true
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>>> > 2013-02-28 02:07:51,925 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>>>> true
>>>> > 2013-02-28 02:07:52,462 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>>>> occuring
>>>> > more than 10 times
>>>> > 2013-02-28 02:07:52,466 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>>> > 2013-02-28 02:07:52,467 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> > dfs.namenode.safemode.min.datanodes = 0
>>>> > 2013-02-28 02:07:52,467 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> > dfs.namenode.safemode.extension = 0
>>>> > 2013-02-28 02:07:52,469 INFO
>>>> org.apache.hadoop.hdfs.server.common.Storage:
>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>>>> > 2013-02-28 02:07:52,471 INFO
>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>>>> metrics
>>>> > system...
>>>> > 2013-02-28 02:07:52,472 INFO
>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>> system
>>>> > stopped.
>>>> > 2013-02-28 02:07:52,473 INFO
>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>> system
>>>> > shutdown complete.
>>>> > 2013-02-28 02:07:52,473 FATAL
>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>>>> namenode join
>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>> Directory
>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
>>>> storage
>>>> > directory does not exist or is not accessible.
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>>> with
>>>> > status 1
>>>> > 2013-02-28 02:08:48,913 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>>> > /************************************************************
>>>> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
>>>> 192.168.1.3
>>>> >
>>>> >
>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>>>> > <pr...@gmail.com> wrote:
>>>> >>
>>>> >> Hi Guys,
>>>> >>
>>>> >> I have space on other partition. Can I change the path for cache
>>>> files on
>>>> >> other partition ? I have below properties . Can it resolve the issue
>>>> ? If i
>>>> >> change the path to other directories and restart services I get the
>>>> below
>>>> >> error while starting the service namenode. I didn't find anything in
>>>> logs so
>>>> >> far. Can you please suggest something ?
>>>> >>
>>>> >> <property>
>>>> >> <name>hadoop.tmp.dir</name>
>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>>>> >> </property>
>>>> >> <property>
>>>> >> <name>dfs.namenode.name.dir</name>
>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>>>> >> </property>
>>>> >> <property>
>>>> >> <name>dfs.namenode.checkpoint.dir</name>
>>>> >>
>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>>>> }/dfs/namesecondary</value>
>>>> >> </property>
>>>> >> <property>
>>>> >>
>>>> >>
>>>> >> Service namenode is failing
>>>> >>
>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
>>>> status; done
>>>> >> Hadoop datanode is running [ OK ]
>>>> >> Hadoop namenode is dead and pid file exists [FAILED]
>>>> >> Hadoop secondarynamenode is running [ OK ]
>>>> >>
>>>> >> Thanks,
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>>>> >> <pr...@gmail.com> wrote:
>>>> >>>
>>>> >>>
>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>>>> wrote:
>>>> >>>>
>>>> >>>> http://NNHOST:50070/conf
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> Harsh, I changed the value as said & restarted service NN. For
>>>> verifying
>>>> >>> i checked the http link that you gave and i saw the property their
>>>> but on
>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>>>> missing
>>>> >>> blocks. Please check the logs or run fsck in order to identify the
>>>> missing
>>>> >>> blocks.) when i clicked on this link i can see file names . Do I
>>>> need to
>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>>>> command fsck
>>>> >>> that i can run on the running hadoop ?
>>>> >>>
>>>> >>> Thanks,
>>>> >>>
>>>> >>
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>
>
Re: NameNode low on available disk space
Posted by shashwat shriparv <dw...@gmail.com>.
And alos check for this parameter
dfs.namenode.safemode.threshold-pct
check the file system if health status is correct.
and if underreplication is there??
∞
Shashwat Shriparv
On Thu, Feb 28, 2013 at 5:55 PM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
>
> On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> mpl: Stopping NameNo
>
>
> Please check how disk space uses is fluctuation while running MR Jobs.
>
>
>
> ∞
> Shashwat Shriparv
>
>
Re: NameNode low on available disk space
Posted by shashwat shriparv <dw...@gmail.com>.
And alos check for this parameter
dfs.namenode.safemode.threshold-pct
check the file system if health status is correct.
and if underreplication is there??
∞
Shashwat Shriparv
On Thu, Feb 28, 2013 at 5:55 PM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
>
> On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> mpl: Stopping NameNo
>
>
> Please check how disk space uses is fluctuation while running MR Jobs.
>
>
>
> ∞
> Shashwat Shriparv
>
>
Re: NameNode low on available disk space
Posted by shashwat shriparv <dw...@gmail.com>.
And alos check for this parameter
dfs.namenode.safemode.threshold-pct
check the file system if health status is correct.
and if underreplication is there??
∞
Shashwat Shriparv
On Thu, Feb 28, 2013 at 5:55 PM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
>
> On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> mpl: Stopping NameNo
>
>
> Please check how disk space uses is fluctuation while running MR Jobs.
>
>
>
> ∞
> Shashwat Shriparv
>
>
Re: NameNode low on available disk space
Posted by shashwat shriparv <dw...@gmail.com>.
And alos check for this parameter
dfs.namenode.safemode.threshold-pct
check the file system if health status is correct.
and if underreplication is there??
∞
Shashwat Shriparv
On Thu, Feb 28, 2013 at 5:55 PM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
>
> On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> mpl: Stopping NameNo
>
>
> Please check how disk space uses is fluctuation while running MR Jobs.
>
>
>
> ∞
> Shashwat Shriparv
>
>
Re: NameNode low on available disk space
Posted by shashwat shriparv <dw...@gmail.com>.
On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> mpl: Stopping NameNo
Please check how disk space uses is fluctuation while running MR Jobs.
∞
Shashwat Shriparv
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
Hi,
May i know the answer plz ?
Thanks,
On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> Even i created the
> file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock and set permission .
> when i restart hadoop services. It removes and I find below logs.
>
> Do I need to format the NN?
> Below is the command to format the NN ?
> Any kind of loss while formatting ?
> Is there any way to avoid formatting and change the cache path ?
>
> 2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
> 81133@OPERA-MAST1.ny.os.local
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 05:57:50,905 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> java.io.IOException: NameNode is not formatted.
>
> Command to format the NN.
>
> sudo -u hdfs hdfs namenode -format
>
> Thanks,
>
>
> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> After creating the directory and setting permission I tried to restart
>> the services and i get error
>> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> 7275@OPERA-MAST1.ny.os.local" and services are not being started.
>>
>> Need to check few logs from below logs.
>> ===================================
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack of redundant sto
>> rage directories!
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> dataloss due to lack of re
>> dundant storage directories!
>>
>>
>> ************************************************************/
>> 2013-02-28 05:06:23,385 WARN
>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
>> 2013-02-28 05:06:23,556 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2013-02-28 05:06:23,556 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> started
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack of redundant sto
>> rage directories!
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> dataloss due to lack of re
>> dundant storage directories!
>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
>> Refreshing hosts (include/exclude) list
>> 2013-02-28 05:06:25,623 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> dfs.block.invalidate.limit=1000
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> dfs.block.access.token.enable=false
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> defaultReplication = 1
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
>> = 512
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
>> = 1
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplicationStreams = 2
>> 2013-02-28 05:06:26,016 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> shouldCheckForEnoughRacks = false
>> 2013-02-28 05:06:26,016 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> replicationRecheckInterval = 3000
>> 2013-02-28 05:06:26,016 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> encryptDataTransfer = false
>> 2013-02-28 05:06:26,022 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
>> hdfs (auth:SIMPLE)
>> 2013-02-28 05:06:26,022 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
>> hadmin
>> 2013-02-28 05:06:26,022 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
>> true
>> 2013-02-28 05:06:26,023 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>> 2013-02-28 05:06:26,026 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>> 2013-02-28 05:06:26,359 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring more than 10 times
>> 2013-02-28 05:06:26,361 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> 2013-02-28 05:06:26,361 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.min.datanodes = 0
>> 2013-02-28 05:06:26,361 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.extension = 0
>> 2013-02-28 05:06:26,378 INFO
>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> 7275@OPERA-MAST1.ny.os.local
>> 2013-02-28 05:06:26,381 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics system...
>> 2013-02-28 05:06:26,381 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> stopped.
>> 2013-02-28 05:06:26,381 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> shutdown complete.
>> 2013-02-28 05:06:26,382 FATAL
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>> java.io.IOException: NameNode is not formatted.
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> with status 1
>> 2013-02-28 05:06:26,394 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>>
>>
>>
>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera <
>> project.linux.proj@gmail.com> wrote:
>>
>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created
>>> . If I do compare with the older path the permissions are same on the
>>> parent directories.
>>> Do I need to create this this directory manually and set the permission ?
>>>
>>> Older Path
>>>
>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>>> total 4
>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>>>
>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>>> total 12
>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>
>>>
>>> New Path
>>>
>>> # ll /mnt/san1/hdfs/cache/hdfs/
>>> total 4
>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>>>
>>>
>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>>> total 4
>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>
>>>
>>> Thanks,
>>>
>>>
>>>
>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> The exact error is displayed on your log and should be somewhat self
>>>> explanatory:
>>>>
>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>>>> state: storage directory does not exist or is not accessible.
>>>>
>>>> Please check this one's availability, permissions (the NN user should
>>>> be able to access it).
>>>>
>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>>>> <pr...@gmail.com> wrote:
>>>> > Please find below logs for shutting down the namenode service. Can
>>>> anybody
>>>> > check this
>>>> >
>>>> > 2013-02-28 02:07:51,752 WARN
>>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>> > configuration files. Please update hdfs configuration.
>>>> > 2013-02-28 02:07:51,754 WARN
>>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>> > configuration files. Please update hdfs configuration.
>>>> > 2013-02-28 02:07:51,754 WARN
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>>>> storage
>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>>>> to lack
>>>> > of redundant storage directories!
>>>> > 2013-02-28 02:07:51,754 WARN
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>>>> namespace
>>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>>> > dataloss due to lack of redundant storage directories!
>>>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>>>> > Refreshing hosts (include/exclude) list
>>>> > 2013-02-28 02:07:51,890 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>>> > dfs.block.invalidate.limit=1000
>>>> > 2013-02-28 02:07:51,909 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > dfs.block.access.token.enable=false
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > defaultReplication = 1
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> maxReplication
>>>> > = 512
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> minReplication
>>>> > = 1
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > maxReplicationStreams = 2
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > shouldCheckForEnoughRacks = false
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > replicationRecheckInterval = 3000
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > encryptDataTransfer = false
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>>>> =
>>>> > hdfs (auth:SIMPLE)
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>>>> =
>>>> > hadmin
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> isPermissionEnabled =
>>>> > true
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>>> > 2013-02-28 02:07:51,925 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>>>> true
>>>> > 2013-02-28 02:07:52,462 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>>>> occuring
>>>> > more than 10 times
>>>> > 2013-02-28 02:07:52,466 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>>> > 2013-02-28 02:07:52,467 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> > dfs.namenode.safemode.min.datanodes = 0
>>>> > 2013-02-28 02:07:52,467 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> > dfs.namenode.safemode.extension = 0
>>>> > 2013-02-28 02:07:52,469 INFO
>>>> org.apache.hadoop.hdfs.server.common.Storage:
>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>>>> > 2013-02-28 02:07:52,471 INFO
>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>>>> metrics
>>>> > system...
>>>> > 2013-02-28 02:07:52,472 INFO
>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>> system
>>>> > stopped.
>>>> > 2013-02-28 02:07:52,473 INFO
>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>> system
>>>> > shutdown complete.
>>>> > 2013-02-28 02:07:52,473 FATAL
>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>>>> namenode join
>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>> Directory
>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
>>>> storage
>>>> > directory does not exist or is not accessible.
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>>> with
>>>> > status 1
>>>> > 2013-02-28 02:08:48,913 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>>> > /************************************************************
>>>> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
>>>> 192.168.1.3
>>>> >
>>>> >
>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>>>> > <pr...@gmail.com> wrote:
>>>> >>
>>>> >> Hi Guys,
>>>> >>
>>>> >> I have space on other partition. Can I change the path for cache
>>>> files on
>>>> >> other partition ? I have below properties . Can it resolve the issue
>>>> ? If i
>>>> >> change the path to other directories and restart services I get the
>>>> below
>>>> >> error while starting the service namenode. I didn't find anything in
>>>> logs so
>>>> >> far. Can you please suggest something ?
>>>> >>
>>>> >> <property>
>>>> >> <name>hadoop.tmp.dir</name>
>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>>>> >> </property>
>>>> >> <property>
>>>> >> <name>dfs.namenode.name.dir</name>
>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>>>> >> </property>
>>>> >> <property>
>>>> >> <name>dfs.namenode.checkpoint.dir</name>
>>>> >>
>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>>>> }/dfs/namesecondary</value>
>>>> >> </property>
>>>> >> <property>
>>>> >>
>>>> >>
>>>> >> Service namenode is failing
>>>> >>
>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
>>>> status; done
>>>> >> Hadoop datanode is running [ OK ]
>>>> >> Hadoop namenode is dead and pid file exists [FAILED]
>>>> >> Hadoop secondarynamenode is running [ OK ]
>>>> >>
>>>> >> Thanks,
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>>>> >> <pr...@gmail.com> wrote:
>>>> >>>
>>>> >>>
>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>>>> wrote:
>>>> >>>>
>>>> >>>> http://NNHOST:50070/conf
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> Harsh, I changed the value as said & restarted service NN. For
>>>> verifying
>>>> >>> i checked the http link that you gave and i saw the property their
>>>> but on
>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>>>> missing
>>>> >>> blocks. Please check the logs or run fsck in order to identify the
>>>> missing
>>>> >>> blocks.) when i clicked on this link i can see file names . Do I
>>>> need to
>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>>>> command fsck
>>>> >>> that i can run on the running hadoop ?
>>>> >>>
>>>> >>> Thanks,
>>>> >>>
>>>> >>
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
Hi,
May i know the answer plz ?
Thanks,
On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> Even i created the
> file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock and set permission .
> when i restart hadoop services. It removes and I find below logs.
>
> Do I need to format the NN?
> Below is the command to format the NN ?
> Any kind of loss while formatting ?
> Is there any way to avoid formatting and change the cache path ?
>
> 2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
> 81133@OPERA-MAST1.ny.os.local
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 05:57:50,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 05:57:50,905 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> java.io.IOException: NameNode is not formatted.
>
> Command to format the NN.
>
> sudo -u hdfs hdfs namenode -format
>
> Thanks,
>
>
> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> After creating the directory and setting permission I tried to restart
>> the services and i get error
>> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> 7275@OPERA-MAST1.ny.os.local" and services are not being started.
>>
>> Need to check few logs from below logs.
>> ===================================
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack of redundant sto
>> rage directories!
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> dataloss due to lack of re
>> dundant storage directories!
>>
>>
>> ************************************************************/
>> 2013-02-28 05:06:23,385 WARN
>> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
>> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
>> 2013-02-28 05:06:23,556 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2013-02-28 05:06:23,556 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> started
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> configuration files. Please update hdfs configuration.
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack of redundant sto
>> rage directories!
>> 2013-02-28 05:06:24,906 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> dataloss due to lack of re
>> dundant storage directories!
>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
>> Refreshing hosts (include/exclude) list
>> 2013-02-28 05:06:25,623 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> dfs.block.invalidate.limit=1000
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> dfs.block.access.token.enable=false
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> defaultReplication = 1
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
>> = 512
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
>> = 1
>> 2013-02-28 05:06:26,015 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplicationStreams = 2
>> 2013-02-28 05:06:26,016 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> shouldCheckForEnoughRacks = false
>> 2013-02-28 05:06:26,016 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> replicationRecheckInterval = 3000
>> 2013-02-28 05:06:26,016 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> encryptDataTransfer = false
>> 2013-02-28 05:06:26,022 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
>> hdfs (auth:SIMPLE)
>> 2013-02-28 05:06:26,022 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
>> hadmin
>> 2013-02-28 05:06:26,022 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
>> true
>> 2013-02-28 05:06:26,023 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>> 2013-02-28 05:06:26,026 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>> 2013-02-28 05:06:26,359 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring more than 10 times
>> 2013-02-28 05:06:26,361 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> 2013-02-28 05:06:26,361 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.min.datanodes = 0
>> 2013-02-28 05:06:26,361 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.extension = 0
>> 2013-02-28 05:06:26,378 INFO
>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
>> 7275@OPERA-MAST1.ny.os.local
>> 2013-02-28 05:06:26,381 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics system...
>> 2013-02-28 05:06:26,381 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> stopped.
>> 2013-02-28 05:06:26,381 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> shutdown complete.
>> 2013-02-28 05:06:26,382 FATAL
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>> java.io.IOException: NameNode is not formatted.
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> with status 1
>> 2013-02-28 05:06:26,394 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>>
>>
>>
>> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera <
>> project.linux.proj@gmail.com> wrote:
>>
>>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created
>>> . If I do compare with the older path the permissions are same on the
>>> parent directories.
>>> Do I need to create this this directory manually and set the permission ?
>>>
>>> Older Path
>>>
>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>>> total 4
>>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>>>
>>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>>> total 12
>>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>
>>>
>>> New Path
>>>
>>> # ll /mnt/san1/hdfs/cache/hdfs/
>>> total 4
>>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>>>
>>>
>>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>>> total 4
>>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>>
>>>
>>> Thanks,
>>>
>>>
>>>
>>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> The exact error is displayed on your log and should be somewhat self
>>>> explanatory:
>>>>
>>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>>>> state: storage directory does not exist or is not accessible.
>>>>
>>>> Please check this one's availability, permissions (the NN user should
>>>> be able to access it).
>>>>
>>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>>>> <pr...@gmail.com> wrote:
>>>> > Please find below logs for shutting down the namenode service. Can
>>>> anybody
>>>> > check this
>>>> >
>>>> > 2013-02-28 02:07:51,752 WARN
>>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>> > configuration files. Please update hdfs configuration.
>>>> > 2013-02-28 02:07:51,754 WARN
>>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>>> > configuration files. Please update hdfs configuration.
>>>> > 2013-02-28 02:07:51,754 WARN
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>>>> storage
>>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>>>> to lack
>>>> > of redundant storage directories!
>>>> > 2013-02-28 02:07:51,754 WARN
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
>>>> namespace
>>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>>> > dataloss due to lack of redundant storage directories!
>>>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>>>> > Refreshing hosts (include/exclude) list
>>>> > 2013-02-28 02:07:51,890 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>>> > dfs.block.invalidate.limit=1000
>>>> > 2013-02-28 02:07:51,909 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > dfs.block.access.token.enable=false
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > defaultReplication = 1
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> maxReplication
>>>> > = 512
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> minReplication
>>>> > = 1
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > maxReplicationStreams = 2
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > shouldCheckForEnoughRacks = false
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > replicationRecheckInterval = 3000
>>>> > 2013-02-28 02:07:51,910 INFO
>>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>>> > encryptDataTransfer = false
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>>>> =
>>>> > hdfs (auth:SIMPLE)
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>>>> =
>>>> > hadmin
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> isPermissionEnabled =
>>>> > true
>>>> > 2013-02-28 02:07:51,920 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>>> > 2013-02-28 02:07:51,925 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>>>> true
>>>> > 2013-02-28 02:07:52,462 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>>>> occuring
>>>> > more than 10 times
>>>> > 2013-02-28 02:07:52,466 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>>> > 2013-02-28 02:07:52,467 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> > dfs.namenode.safemode.min.datanodes = 0
>>>> > 2013-02-28 02:07:52,467 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>> > dfs.namenode.safemode.extension = 0
>>>> > 2013-02-28 02:07:52,469 INFO
>>>> org.apache.hadoop.hdfs.server.common.Storage:
>>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>>>> > 2013-02-28 02:07:52,471 INFO
>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>>>> metrics
>>>> > system...
>>>> > 2013-02-28 02:07:52,472 INFO
>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>> system
>>>> > stopped.
>>>> > 2013-02-28 02:07:52,473 INFO
>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>> system
>>>> > shutdown complete.
>>>> > 2013-02-28 02:07:52,473 FATAL
>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
>>>> namenode join
>>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>>> Directory
>>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state:
>>>> storage
>>>> > directory does not exist or is not accessible.
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>>> with
>>>> > status 1
>>>> > 2013-02-28 02:08:48,913 INFO
>>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>>> > /************************************************************
>>>> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
>>>> 192.168.1.3
>>>> >
>>>> >
>>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>>>> > <pr...@gmail.com> wrote:
>>>> >>
>>>> >> Hi Guys,
>>>> >>
>>>> >> I have space on other partition. Can I change the path for cache
>>>> files on
>>>> >> other partition ? I have below properties . Can it resolve the issue
>>>> ? If i
>>>> >> change the path to other directories and restart services I get the
>>>> below
>>>> >> error while starting the service namenode. I didn't find anything in
>>>> logs so
>>>> >> far. Can you please suggest something ?
>>>> >>
>>>> >> <property>
>>>> >> <name>hadoop.tmp.dir</name>
>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>>>> >> </property>
>>>> >> <property>
>>>> >> <name>dfs.namenode.name.dir</name>
>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>>>> >> </property>
>>>> >> <property>
>>>> >> <name>dfs.namenode.checkpoint.dir</name>
>>>> >>
>>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>>>> }/dfs/namesecondary</value>
>>>> >> </property>
>>>> >> <property>
>>>> >>
>>>> >>
>>>> >> Service namenode is failing
>>>> >>
>>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
>>>> status; done
>>>> >> Hadoop datanode is running [ OK ]
>>>> >> Hadoop namenode is dead and pid file exists [FAILED]
>>>> >> Hadoop secondarynamenode is running [ OK ]
>>>> >>
>>>> >> Thanks,
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>>>> >> <pr...@gmail.com> wrote:
>>>> >>>
>>>> >>>
>>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>>>> wrote:
>>>> >>>>
>>>> >>>> http://NNHOST:50070/conf
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> Harsh, I changed the value as said & restarted service NN. For
>>>> verifying
>>>> >>> i checked the http link that you gave and i saw the property their
>>>> but on
>>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>>>> missing
>>>> >>> blocks. Please check the logs or run fsck in order to identify the
>>>> missing
>>>> >>> blocks.) when i clicked on this link i can see file names . Do I
>>>> need to
>>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>>>> command fsck
>>>> >>> that i can run on the running hadoop ?
>>>> >>>
>>>> >>> Thanks,
>>>> >>>
>>>> >>
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>
>
Re: NameNode low on available disk space
Posted by shashwat shriparv <dw...@gmail.com>.
On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> mpl: Stopping NameNo
Please check how disk space uses is fluctuation while running MR Jobs.
∞
Shashwat Shriparv
Re: NameNode low on available disk space
Posted by shashwat shriparv <dw...@gmail.com>.
On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> mpl: Stopping NameNo
Please check how disk space uses is fluctuation while running MR Jobs.
∞
Shashwat Shriparv
Re: NameNode low on available disk space
Posted by shashwat shriparv <dw...@gmail.com>.
On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> mpl: Stopping NameNo
Please check how disk space uses is fluctuation while running MR Jobs.
∞
Shashwat Shriparv
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock and
set permission . when i restart hadoop services. It removes and I find
below logs.
Do I need to format the NN?
Below is the command to format the NN ?
Any kind of loss while formatting ?
Is there any way to avoid formatting and change the cache path ?
2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
81133@OPERA-MAST1.ny.os.local
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:57:50,905 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
Command to format the NN.
sudo -u hdfs hdfs namenode -format
Thanks,
On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> After creating the directory and setting permission I tried to restart the
> services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are not
> being started.
>
> Need to check few logs from below logs.
> ===================================
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant sto
> rage directories!
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of re
> dundant storage directories!
>
>
> ************************************************************/
> 2013-02-28 05:06:23,385 WARN
> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
> 2013-02-28 05:06:23,556 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-02-28 05:06:23,556 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant sto
> rage directories!
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of re
> dundant storage directories!
> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-02-28 05:06:25,623 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
> = 512
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
> = 1
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
> 2013-02-28 05:06:26,016 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
> 2013-02-28 05:06:26,016 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
> 2013-02-28 05:06:26,016 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
> 2013-02-28 05:06:26,022 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> hdfs (auth:SIMPLE)
> 2013-02-28 05:06:26,022 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> hadmin
> 2013-02-28 05:06:26,022 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
> 2013-02-28 05:06:26,023 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> 2013-02-28 05:06:26,026 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> 2013-02-28 05:06:26,359 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
> 2013-02-28 05:06:26,361 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> 2013-02-28 05:06:26,361 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
> 2013-02-28 05:06:26,361 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 0
> 2013-02-28 05:06:26,378 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
> 7275@OPERA-MAST1.ny.os.local
> 2013-02-28 05:06:26,381 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
> 2013-02-28 05:06:26,381 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 05:06:26,381 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 05:06:26,382 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> java.io.IOException: NameNode is not formatted.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2013-02-28 05:06:26,394 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>
>
>
> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
>> If I do compare with the older path the permissions are same on the parent
>> directories.
>> Do I need to create this this directory manually and set the permission ?
>>
>> Older Path
>>
>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>> total 4
>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>>
>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>> total 12
>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>
>>
>> New Path
>>
>> # ll /mnt/san1/hdfs/cache/hdfs/
>> total 4
>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>>
>>
>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>> total 4
>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>
>>
>> Thanks,
>>
>>
>>
>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Hi,
>>>
>>> The exact error is displayed on your log and should be somewhat self
>>> explanatory:
>>>
>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>>> state: storage directory does not exist or is not accessible.
>>>
>>> Please check this one's availability, permissions (the NN user should
>>> be able to access it).
>>>
>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>>> <pr...@gmail.com> wrote:
>>> > Please find below logs for shutting down the namenode service. Can
>>> anybody
>>> > check this
>>> >
>>> > 2013-02-28 02:07:51,752 WARN
>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> > configuration files. Please update hdfs configuration.
>>> > 2013-02-28 02:07:51,754 WARN
>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> > configuration files. Please update hdfs configuration.
>>> > 2013-02-28 02:07:51,754 WARN
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>>> storage
>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>>> to lack
>>> > of redundant storage directories!
>>> > 2013-02-28 02:07:51,754 WARN
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>> > dataloss due to lack of redundant storage directories!
>>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>>> > Refreshing hosts (include/exclude) list
>>> > 2013-02-28 02:07:51,890 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>> > dfs.block.invalidate.limit=1000
>>> > 2013-02-28 02:07:51,909 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > dfs.block.access.token.enable=false
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > defaultReplication = 1
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> maxReplication
>>> > = 512
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> minReplication
>>> > = 1
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > maxReplicationStreams = 2
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > shouldCheckForEnoughRacks = false
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > replicationRecheckInterval = 3000
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > encryptDataTransfer = false
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>>> =
>>> > hdfs (auth:SIMPLE)
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>>> =
>>> > hadmin
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> isPermissionEnabled =
>>> > true
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>> > 2013-02-28 02:07:51,925 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>>> true
>>> > 2013-02-28 02:07:52,462 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>>> occuring
>>> > more than 10 times
>>> > 2013-02-28 02:07:52,466 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>> > 2013-02-28 02:07:52,467 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> > dfs.namenode.safemode.min.datanodes = 0
>>> > 2013-02-28 02:07:52,467 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> > dfs.namenode.safemode.extension = 0
>>> > 2013-02-28 02:07:52,469 INFO
>>> org.apache.hadoop.hdfs.server.common.Storage:
>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>>> > 2013-02-28 02:07:52,471 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>>> metrics
>>> > system...
>>> > 2013-02-28 02:07:52,472 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>> system
>>> > stopped.
>>> > 2013-02-28 02:07:52,473 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>> system
>>> > shutdown complete.
>>> > 2013-02-28 02:07:52,473 FATAL
>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
>>> join
>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>> Directory
>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
>>> > directory does not exist or is not accessible.
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>> with
>>> > status 1
>>> > 2013-02-28 02:08:48,913 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>> > /************************************************************
>>> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
>>> 192.168.1.3
>>> >
>>> >
>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>>> > <pr...@gmail.com> wrote:
>>> >>
>>> >> Hi Guys,
>>> >>
>>> >> I have space on other partition. Can I change the path for cache
>>> files on
>>> >> other partition ? I have below properties . Can it resolve the issue
>>> ? If i
>>> >> change the path to other directories and restart services I get the
>>> below
>>> >> error while starting the service namenode. I didn't find anything in
>>> logs so
>>> >> far. Can you please suggest something ?
>>> >>
>>> >> <property>
>>> >> <name>hadoop.tmp.dir</name>
>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>>> >> </property>
>>> >> <property>
>>> >> <name>dfs.namenode.name.dir</name>
>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>>> >> </property>
>>> >> <property>
>>> >> <name>dfs.namenode.checkpoint.dir</name>
>>> >>
>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>>> }/dfs/namesecondary</value>
>>> >> </property>
>>> >> <property>
>>> >>
>>> >>
>>> >> Service namenode is failing
>>> >>
>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
>>> done
>>> >> Hadoop datanode is running [ OK ]
>>> >> Hadoop namenode is dead and pid file exists [FAILED]
>>> >> Hadoop secondarynamenode is running [ OK ]
>>> >>
>>> >> Thanks,
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>>> >> <pr...@gmail.com> wrote:
>>> >>>
>>> >>>
>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>>> wrote:
>>> >>>>
>>> >>>> http://NNHOST:50070/conf
>>> >>>
>>> >>>
>>> >>>
>>> >>> Harsh, I changed the value as said & restarted service NN. For
>>> verifying
>>> >>> i checked the http link that you gave and i saw the property their
>>> but on
>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>>> missing
>>> >>> blocks. Please check the logs or run fsck in order to identify the
>>> missing
>>> >>> blocks.) when i clicked on this link i can see file names . Do I
>>> need to
>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>>> command fsck
>>> >>> that i can run on the running hadoop ?
>>> >>>
>>> >>> Thanks,
>>> >>>
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock and
set permission . when i restart hadoop services. It removes and I find
below logs.
Do I need to format the NN?
Below is the command to format the NN ?
Any kind of loss while formatting ?
Is there any way to avoid formatting and change the cache path ?
2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
81133@OPERA-MAST1.ny.os.local
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:57:50,905 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
Command to format the NN.
sudo -u hdfs hdfs namenode -format
Thanks,
On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> After creating the directory and setting permission I tried to restart the
> services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are not
> being started.
>
> Need to check few logs from below logs.
> ===================================
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant sto
> rage directories!
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of re
> dundant storage directories!
>
>
> ************************************************************/
> 2013-02-28 05:06:23,385 WARN
> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
> 2013-02-28 05:06:23,556 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-02-28 05:06:23,556 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant sto
> rage directories!
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of re
> dundant storage directories!
> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-02-28 05:06:25,623 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
> = 512
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
> = 1
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
> 2013-02-28 05:06:26,016 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
> 2013-02-28 05:06:26,016 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
> 2013-02-28 05:06:26,016 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
> 2013-02-28 05:06:26,022 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> hdfs (auth:SIMPLE)
> 2013-02-28 05:06:26,022 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> hadmin
> 2013-02-28 05:06:26,022 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
> 2013-02-28 05:06:26,023 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> 2013-02-28 05:06:26,026 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> 2013-02-28 05:06:26,359 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
> 2013-02-28 05:06:26,361 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> 2013-02-28 05:06:26,361 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
> 2013-02-28 05:06:26,361 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 0
> 2013-02-28 05:06:26,378 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
> 7275@OPERA-MAST1.ny.os.local
> 2013-02-28 05:06:26,381 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
> 2013-02-28 05:06:26,381 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 05:06:26,381 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 05:06:26,382 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> java.io.IOException: NameNode is not formatted.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2013-02-28 05:06:26,394 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>
>
>
> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
>> If I do compare with the older path the permissions are same on the parent
>> directories.
>> Do I need to create this this directory manually and set the permission ?
>>
>> Older Path
>>
>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>> total 4
>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>>
>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>> total 12
>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>
>>
>> New Path
>>
>> # ll /mnt/san1/hdfs/cache/hdfs/
>> total 4
>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>>
>>
>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>> total 4
>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>
>>
>> Thanks,
>>
>>
>>
>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Hi,
>>>
>>> The exact error is displayed on your log and should be somewhat self
>>> explanatory:
>>>
>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>>> state: storage directory does not exist or is not accessible.
>>>
>>> Please check this one's availability, permissions (the NN user should
>>> be able to access it).
>>>
>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>>> <pr...@gmail.com> wrote:
>>> > Please find below logs for shutting down the namenode service. Can
>>> anybody
>>> > check this
>>> >
>>> > 2013-02-28 02:07:51,752 WARN
>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> > configuration files. Please update hdfs configuration.
>>> > 2013-02-28 02:07:51,754 WARN
>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> > configuration files. Please update hdfs configuration.
>>> > 2013-02-28 02:07:51,754 WARN
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>>> storage
>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>>> to lack
>>> > of redundant storage directories!
>>> > 2013-02-28 02:07:51,754 WARN
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>> > dataloss due to lack of redundant storage directories!
>>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>>> > Refreshing hosts (include/exclude) list
>>> > 2013-02-28 02:07:51,890 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>> > dfs.block.invalidate.limit=1000
>>> > 2013-02-28 02:07:51,909 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > dfs.block.access.token.enable=false
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > defaultReplication = 1
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> maxReplication
>>> > = 512
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> minReplication
>>> > = 1
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > maxReplicationStreams = 2
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > shouldCheckForEnoughRacks = false
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > replicationRecheckInterval = 3000
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > encryptDataTransfer = false
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>>> =
>>> > hdfs (auth:SIMPLE)
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>>> =
>>> > hadmin
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> isPermissionEnabled =
>>> > true
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>> > 2013-02-28 02:07:51,925 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>>> true
>>> > 2013-02-28 02:07:52,462 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>>> occuring
>>> > more than 10 times
>>> > 2013-02-28 02:07:52,466 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>> > 2013-02-28 02:07:52,467 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> > dfs.namenode.safemode.min.datanodes = 0
>>> > 2013-02-28 02:07:52,467 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> > dfs.namenode.safemode.extension = 0
>>> > 2013-02-28 02:07:52,469 INFO
>>> org.apache.hadoop.hdfs.server.common.Storage:
>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>>> > 2013-02-28 02:07:52,471 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>>> metrics
>>> > system...
>>> > 2013-02-28 02:07:52,472 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>> system
>>> > stopped.
>>> > 2013-02-28 02:07:52,473 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>> system
>>> > shutdown complete.
>>> > 2013-02-28 02:07:52,473 FATAL
>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
>>> join
>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>> Directory
>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
>>> > directory does not exist or is not accessible.
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>> with
>>> > status 1
>>> > 2013-02-28 02:08:48,913 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>> > /************************************************************
>>> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
>>> 192.168.1.3
>>> >
>>> >
>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>>> > <pr...@gmail.com> wrote:
>>> >>
>>> >> Hi Guys,
>>> >>
>>> >> I have space on other partition. Can I change the path for cache
>>> files on
>>> >> other partition ? I have below properties . Can it resolve the issue
>>> ? If i
>>> >> change the path to other directories and restart services I get the
>>> below
>>> >> error while starting the service namenode. I didn't find anything in
>>> logs so
>>> >> far. Can you please suggest something ?
>>> >>
>>> >> <property>
>>> >> <name>hadoop.tmp.dir</name>
>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>>> >> </property>
>>> >> <property>
>>> >> <name>dfs.namenode.name.dir</name>
>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>>> >> </property>
>>> >> <property>
>>> >> <name>dfs.namenode.checkpoint.dir</name>
>>> >>
>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>>> }/dfs/namesecondary</value>
>>> >> </property>
>>> >> <property>
>>> >>
>>> >>
>>> >> Service namenode is failing
>>> >>
>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
>>> done
>>> >> Hadoop datanode is running [ OK ]
>>> >> Hadoop namenode is dead and pid file exists [FAILED]
>>> >> Hadoop secondarynamenode is running [ OK ]
>>> >>
>>> >> Thanks,
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>>> >> <pr...@gmail.com> wrote:
>>> >>>
>>> >>>
>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>>> wrote:
>>> >>>>
>>> >>>> http://NNHOST:50070/conf
>>> >>>
>>> >>>
>>> >>>
>>> >>> Harsh, I changed the value as said & restarted service NN. For
>>> verifying
>>> >>> i checked the http link that you gave and i saw the property their
>>> but on
>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>>> missing
>>> >>> blocks. Please check the logs or run fsck in order to identify the
>>> missing
>>> >>> blocks.) when i clicked on this link i can see file names . Do I
>>> need to
>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>>> command fsck
>>> >>> that i can run on the running hadoop ?
>>> >>>
>>> >>> Thanks,
>>> >>>
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock and
set permission . when i restart hadoop services. It removes and I find
below logs.
Do I need to format the NN?
Below is the command to format the NN ?
Any kind of loss while formatting ?
Is there any way to avoid formatting and change the cache path ?
2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
81133@OPERA-MAST1.ny.os.local
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:57:50,905 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
Command to format the NN.
sudo -u hdfs hdfs namenode -format
Thanks,
On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> After creating the directory and setting permission I tried to restart the
> services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are not
> being started.
>
> Need to check few logs from below logs.
> ===================================
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant sto
> rage directories!
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of re
> dundant storage directories!
>
>
> ************************************************************/
> 2013-02-28 05:06:23,385 WARN
> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
> 2013-02-28 05:06:23,556 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-02-28 05:06:23,556 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant sto
> rage directories!
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of re
> dundant storage directories!
> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-02-28 05:06:25,623 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
> = 512
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
> = 1
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
> 2013-02-28 05:06:26,016 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
> 2013-02-28 05:06:26,016 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
> 2013-02-28 05:06:26,016 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
> 2013-02-28 05:06:26,022 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> hdfs (auth:SIMPLE)
> 2013-02-28 05:06:26,022 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> hadmin
> 2013-02-28 05:06:26,022 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
> 2013-02-28 05:06:26,023 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> 2013-02-28 05:06:26,026 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> 2013-02-28 05:06:26,359 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
> 2013-02-28 05:06:26,361 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> 2013-02-28 05:06:26,361 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
> 2013-02-28 05:06:26,361 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 0
> 2013-02-28 05:06:26,378 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
> 7275@OPERA-MAST1.ny.os.local
> 2013-02-28 05:06:26,381 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
> 2013-02-28 05:06:26,381 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 05:06:26,381 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 05:06:26,382 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> java.io.IOException: NameNode is not formatted.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2013-02-28 05:06:26,394 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>
>
>
> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
>> If I do compare with the older path the permissions are same on the parent
>> directories.
>> Do I need to create this this directory manually and set the permission ?
>>
>> Older Path
>>
>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>> total 4
>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>>
>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>> total 12
>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>
>>
>> New Path
>>
>> # ll /mnt/san1/hdfs/cache/hdfs/
>> total 4
>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>>
>>
>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>> total 4
>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>
>>
>> Thanks,
>>
>>
>>
>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Hi,
>>>
>>> The exact error is displayed on your log and should be somewhat self
>>> explanatory:
>>>
>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>>> state: storage directory does not exist or is not accessible.
>>>
>>> Please check this one's availability, permissions (the NN user should
>>> be able to access it).
>>>
>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>>> <pr...@gmail.com> wrote:
>>> > Please find below logs for shutting down the namenode service. Can
>>> anybody
>>> > check this
>>> >
>>> > 2013-02-28 02:07:51,752 WARN
>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> > configuration files. Please update hdfs configuration.
>>> > 2013-02-28 02:07:51,754 WARN
>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> > configuration files. Please update hdfs configuration.
>>> > 2013-02-28 02:07:51,754 WARN
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>>> storage
>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>>> to lack
>>> > of redundant storage directories!
>>> > 2013-02-28 02:07:51,754 WARN
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>> > dataloss due to lack of redundant storage directories!
>>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>>> > Refreshing hosts (include/exclude) list
>>> > 2013-02-28 02:07:51,890 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>> > dfs.block.invalidate.limit=1000
>>> > 2013-02-28 02:07:51,909 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > dfs.block.access.token.enable=false
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > defaultReplication = 1
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> maxReplication
>>> > = 512
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> minReplication
>>> > = 1
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > maxReplicationStreams = 2
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > shouldCheckForEnoughRacks = false
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > replicationRecheckInterval = 3000
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > encryptDataTransfer = false
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>>> =
>>> > hdfs (auth:SIMPLE)
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>>> =
>>> > hadmin
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> isPermissionEnabled =
>>> > true
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>> > 2013-02-28 02:07:51,925 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>>> true
>>> > 2013-02-28 02:07:52,462 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>>> occuring
>>> > more than 10 times
>>> > 2013-02-28 02:07:52,466 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>> > 2013-02-28 02:07:52,467 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> > dfs.namenode.safemode.min.datanodes = 0
>>> > 2013-02-28 02:07:52,467 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> > dfs.namenode.safemode.extension = 0
>>> > 2013-02-28 02:07:52,469 INFO
>>> org.apache.hadoop.hdfs.server.common.Storage:
>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>>> > 2013-02-28 02:07:52,471 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>>> metrics
>>> > system...
>>> > 2013-02-28 02:07:52,472 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>> system
>>> > stopped.
>>> > 2013-02-28 02:07:52,473 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>> system
>>> > shutdown complete.
>>> > 2013-02-28 02:07:52,473 FATAL
>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
>>> join
>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>> Directory
>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
>>> > directory does not exist or is not accessible.
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>> with
>>> > status 1
>>> > 2013-02-28 02:08:48,913 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>> > /************************************************************
>>> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
>>> 192.168.1.3
>>> >
>>> >
>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>>> > <pr...@gmail.com> wrote:
>>> >>
>>> >> Hi Guys,
>>> >>
>>> >> I have space on other partition. Can I change the path for cache
>>> files on
>>> >> other partition ? I have below properties . Can it resolve the issue
>>> ? If i
>>> >> change the path to other directories and restart services I get the
>>> below
>>> >> error while starting the service namenode. I didn't find anything in
>>> logs so
>>> >> far. Can you please suggest something ?
>>> >>
>>> >> <property>
>>> >> <name>hadoop.tmp.dir</name>
>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>>> >> </property>
>>> >> <property>
>>> >> <name>dfs.namenode.name.dir</name>
>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>>> >> </property>
>>> >> <property>
>>> >> <name>dfs.namenode.checkpoint.dir</name>
>>> >>
>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>>> }/dfs/namesecondary</value>
>>> >> </property>
>>> >> <property>
>>> >>
>>> >>
>>> >> Service namenode is failing
>>> >>
>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
>>> done
>>> >> Hadoop datanode is running [ OK ]
>>> >> Hadoop namenode is dead and pid file exists [FAILED]
>>> >> Hadoop secondarynamenode is running [ OK ]
>>> >>
>>> >> Thanks,
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>>> >> <pr...@gmail.com> wrote:
>>> >>>
>>> >>>
>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>>> wrote:
>>> >>>>
>>> >>>> http://NNHOST:50070/conf
>>> >>>
>>> >>>
>>> >>>
>>> >>> Harsh, I changed the value as said & restarted service NN. For
>>> verifying
>>> >>> i checked the http link that you gave and i saw the property their
>>> but on
>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>>> missing
>>> >>> blocks. Please check the logs or run fsck in order to identify the
>>> missing
>>> >>> blocks.) when i clicked on this link i can see file names . Do I
>>> need to
>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>>> command fsck
>>> >>> that i can run on the running hadoop ?
>>> >>>
>>> >>> Thanks,
>>> >>>
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock and
set permission . when i restart hadoop services. It removes and I find
below logs.
Do I need to format the NN?
Below is the command to format the NN ?
Any kind of loss while formatting ?
Is there any way to avoid formatting and change the cache path ?
2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
81133@OPERA-MAST1.ny.os.local
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:57:50,905 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
Command to format the NN.
sudo -u hdfs hdfs namenode -format
Thanks,
On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> After creating the directory and setting permission I tried to restart the
> services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
> acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are not
> being started.
>
> Need to check few logs from below logs.
> ===================================
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant sto
> rage directories!
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of re
> dundant storage directories!
>
>
> ************************************************************/
> 2013-02-28 05:06:23,385 WARN
> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
> tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
> 2013-02-28 05:06:23,556 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-02-28 05:06:23,556 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant sto
> rage directories!
> 2013-02-28 05:06:24,906 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of re
> dundant storage directories!
> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-02-28 05:06:25,623 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
> = 512
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
> = 1
> 2013-02-28 05:06:26,015 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
> 2013-02-28 05:06:26,016 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
> 2013-02-28 05:06:26,016 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
> 2013-02-28 05:06:26,016 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
> 2013-02-28 05:06:26,022 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> hdfs (auth:SIMPLE)
> 2013-02-28 05:06:26,022 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> hadmin
> 2013-02-28 05:06:26,022 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
> 2013-02-28 05:06:26,023 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> 2013-02-28 05:06:26,026 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> 2013-02-28 05:06:26,359 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
> 2013-02-28 05:06:26,361 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> 2013-02-28 05:06:26,361 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
> 2013-02-28 05:06:26,361 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 0
> 2013-02-28 05:06:26,378 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
> 7275@OPERA-MAST1.ny.os.local
> 2013-02-28 05:06:26,381 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
> 2013-02-28 05:06:26,381 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 05:06:26,381 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 05:06:26,382 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> java.io.IOException: NameNode is not formatted.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> 2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2013-02-28 05:06:26,394 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
>
>
>
> On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera <
> project.linux.proj@gmail.com> wrote:
>
>> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
>> If I do compare with the older path the permissions are same on the parent
>> directories.
>> Do I need to create this this directory manually and set the permission ?
>>
>> Older Path
>>
>> # ll /var/lib/hadoop-hdfs/cache/hdfs/
>> total 4
>> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>>
>> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
>> total 12
>> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
>> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>
>>
>> New Path
>>
>> # ll /mnt/san1/hdfs/cache/hdfs/
>> total 4
>> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>>
>>
>> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
>> total 4
>> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>>
>>
>> Thanks,
>>
>>
>>
>> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Hi,
>>>
>>> The exact error is displayed on your log and should be somewhat self
>>> explanatory:
>>>
>>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>>> state: storage directory does not exist or is not accessible.
>>>
>>> Please check this one's availability, permissions (the NN user should
>>> be able to access it).
>>>
>>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>>> <pr...@gmail.com> wrote:
>>> > Please find below logs for shutting down the namenode service. Can
>>> anybody
>>> > check this
>>> >
>>> > 2013-02-28 02:07:51,752 WARN
>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> > configuration files. Please update hdfs configuration.
>>> > 2013-02-28 02:07:51,754 WARN
>>> org.apache.hadoop.hdfs.server.common.Util: Path
>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>>> > configuration files. Please update hdfs configuration.
>>> > 2013-02-28 02:07:51,754 WARN
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>>> storage
>>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due
>>> to lack
>>> > of redundant storage directories!
>>> > 2013-02-28 02:07:51,754 WARN
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>>> > dataloss due to lack of redundant storage directories!
>>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>>> > Refreshing hosts (include/exclude) list
>>> > 2013-02-28 02:07:51,890 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>>> > dfs.block.invalidate.limit=1000
>>> > 2013-02-28 02:07:51,909 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > dfs.block.access.token.enable=false
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > defaultReplication = 1
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> maxReplication
>>> > = 512
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> minReplication
>>> > = 1
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > maxReplicationStreams = 2
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > shouldCheckForEnoughRacks = false
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > replicationRecheckInterval = 3000
>>> > 2013-02-28 02:07:51,910 INFO
>>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>>> > encryptDataTransfer = false
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>>> =
>>> > hdfs (auth:SIMPLE)
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>>> =
>>> > hadmin
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> isPermissionEnabled =
>>> > true
>>> > 2013-02-28 02:07:51,920 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>> > 2013-02-28 02:07:51,925 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>>> true
>>> > 2013-02-28 02:07:52,462 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>>> occuring
>>> > more than 10 times
>>> > 2013-02-28 02:07:52,466 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>> > 2013-02-28 02:07:52,467 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> > dfs.namenode.safemode.min.datanodes = 0
>>> > 2013-02-28 02:07:52,467 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>> > dfs.namenode.safemode.extension = 0
>>> > 2013-02-28 02:07:52,469 INFO
>>> org.apache.hadoop.hdfs.server.common.Storage:
>>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>>> > 2013-02-28 02:07:52,471 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>>> metrics
>>> > system...
>>> > 2013-02-28 02:07:52,472 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>> system
>>> > stopped.
>>> > 2013-02-28 02:07:52,473 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>> system
>>> > shutdown complete.
>>> > 2013-02-28 02:07:52,473 FATAL
>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
>>> join
>>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>>> Directory
>>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
>>> > directory does not exist or is not accessible.
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>>> with
>>> > status 1
>>> > 2013-02-28 02:08:48,913 INFO
>>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>> > /************************************************************
>>> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
>>> 192.168.1.3
>>> >
>>> >
>>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>>> > <pr...@gmail.com> wrote:
>>> >>
>>> >> Hi Guys,
>>> >>
>>> >> I have space on other partition. Can I change the path for cache
>>> files on
>>> >> other partition ? I have below properties . Can it resolve the issue
>>> ? If i
>>> >> change the path to other directories and restart services I get the
>>> below
>>> >> error while starting the service namenode. I didn't find anything in
>>> logs so
>>> >> far. Can you please suggest something ?
>>> >>
>>> >> <property>
>>> >> <name>hadoop.tmp.dir</name>
>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>>> >> </property>
>>> >> <property>
>>> >> <name>dfs.namenode.name.dir</name>
>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>>> >> </property>
>>> >> <property>
>>> >> <name>dfs.namenode.checkpoint.dir</name>
>>> >>
>>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>>> }/dfs/namesecondary</value>
>>> >> </property>
>>> >> <property>
>>> >>
>>> >>
>>> >> Service namenode is failing
>>> >>
>>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
>>> done
>>> >> Hadoop datanode is running [ OK ]
>>> >> Hadoop namenode is dead and pid file exists [FAILED]
>>> >> Hadoop secondarynamenode is running [ OK ]
>>> >>
>>> >> Thanks,
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>>> >> <pr...@gmail.com> wrote:
>>> >>>
>>> >>>
>>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com>
>>> wrote:
>>> >>>>
>>> >>>> http://NNHOST:50070/conf
>>> >>>
>>> >>>
>>> >>>
>>> >>> Harsh, I changed the value as said & restarted service NN. For
>>> verifying
>>> >>> i checked the http link that you gave and i saw the property their
>>> but on
>>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4
>>> missing
>>> >>> blocks. Please check the logs or run fsck in order to identify the
>>> missing
>>> >>> blocks.) when i clicked on this link i can see file names . Do I
>>> need to
>>> >>> reboot the machine to run fsck on root fs/ or is there hadoop
>>> command fsck
>>> >>> that i can run on the running hadoop ?
>>> >>>
>>> >>> Thanks,
>>> >>>
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
After creating the directory and setting permission I tried to restart the
services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are not
being started.
Need to check few logs from below logs.
===================================
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
************************************************************/
2013-02-28 05:06:23,385 WARN org.apache.hadoop.metrics2.impl.MetricsConfig:
Cannot locate configuration: tried
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
started
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2013-02-28 05:06:25,623 INFO
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=1000
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.block.access.token.enable=false
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
defaultReplication = 1
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxReplicationStreams = 2
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
shouldCheckForEnoughRacks = false
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
replicationRecheckInterval = 3000
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
encryptDataTransfer = false
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
hdfs (auth:SIMPLE)
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
hadmin
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true
2013-02-28 05:06:26,023 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 05:06:26,026 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 05:06:26,359 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.extension = 0
2013-02-28 05:06:26,378 INFO org.apache.hadoop.hdfs.server.common.Storage:
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
7275@OPERA-MAST1.ny.os.local
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:06:26,382 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2013-02-28 05:06:26,394 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
> If I do compare with the older path the permissions are same on the parent
> directories.
> Do I need to create this this directory manually and set the permission ?
>
> Older Path
>
> # ll /var/lib/hadoop-hdfs/cache/hdfs/
> total 4
> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>
> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
> total 12
> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>
>
> New Path
>
> # ll /mnt/san1/hdfs/cache/hdfs/
> total 4
> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>
>
> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
> total 4
> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>
>
> Thanks,
>
>
>
> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi,
>>
>> The exact error is displayed on your log and should be somewhat self
>> explanatory:
>>
>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>> state: storage directory does not exist or is not accessible.
>>
>> Please check this one's availability, permissions (the NN user should
>> be able to access it).
>>
>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>> <pr...@gmail.com> wrote:
>> > Please find below logs for shutting down the namenode service. Can
>> anybody
>> > check this
>> >
>> > 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path
>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> > configuration files. Please update hdfs configuration.
>> > 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path
>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> > configuration files. Please update hdfs configuration.
>> > 2013-02-28 02:07:51,754 WARN
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>> storage
>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack
>> > of redundant storage directories!
>> > 2013-02-28 02:07:51,754 WARN
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> > dataloss due to lack of redundant storage directories!
>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>> > Refreshing hosts (include/exclude) list
>> > 2013-02-28 02:07:51,890 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> > dfs.block.invalidate.limit=1000
>> > 2013-02-28 02:07:51,909 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > dfs.block.access.token.enable=false
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > defaultReplication = 1
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplication
>> > = 512
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> minReplication
>> > = 1
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > maxReplicationStreams = 2
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > shouldCheckForEnoughRacks = false
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > replicationRecheckInterval = 3000
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > encryptDataTransfer = false
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>> =
>> > hdfs (auth:SIMPLE)
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>> =
>> > hadmin
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> isPermissionEnabled =
>> > true
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>> > 2013-02-28 02:07:51,925 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>> true
>> > 2013-02-28 02:07:52,462 INFO
>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring
>> > more than 10 times
>> > 2013-02-28 02:07:52,466 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> > 2013-02-28 02:07:52,467 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> > dfs.namenode.safemode.min.datanodes = 0
>> > 2013-02-28 02:07:52,467 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> > dfs.namenode.safemode.extension = 0
>> > 2013-02-28 02:07:52,469 INFO
>> org.apache.hadoop.hdfs.server.common.Storage:
>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>> > 2013-02-28 02:07:52,471 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics
>> > system...
>> > 2013-02-28 02:07:52,472 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> > stopped.
>> > 2013-02-28 02:07:52,473 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> > shutdown complete.
>> > 2013-02-28 02:07:52,473 FATAL
>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
>> join
>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> Directory
>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
>> > directory does not exist or is not accessible.
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> > at
>> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> with
>> > status 1
>> > 2013-02-28 02:08:48,913 INFO
>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> > /************************************************************
>> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
>> 192.168.1.3
>> >
>> >
>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>> > <pr...@gmail.com> wrote:
>> >>
>> >> Hi Guys,
>> >>
>> >> I have space on other partition. Can I change the path for cache files
>> on
>> >> other partition ? I have below properties . Can it resolve the issue ?
>> If i
>> >> change the path to other directories and restart services I get the
>> below
>> >> error while starting the service namenode. I didn't find anything in
>> logs so
>> >> far. Can you please suggest something ?
>> >>
>> >> <property>
>> >> <name>hadoop.tmp.dir</name>
>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>> >> </property>
>> >> <property>
>> >> <name>dfs.namenode.name.dir</name>
>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>> >> </property>
>> >> <property>
>> >> <name>dfs.namenode.checkpoint.dir</name>
>> >>
>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/namesecondary</value>
>> >> </property>
>> >> <property>
>> >>
>> >>
>> >> Service namenode is failing
>> >>
>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
>> done
>> >> Hadoop datanode is running [ OK ]
>> >> Hadoop namenode is dead and pid file exists [FAILED]
>> >> Hadoop secondarynamenode is running [ OK ]
>> >>
>> >> Thanks,
>> >>
>> >>
>> >>
>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>> >> <pr...@gmail.com> wrote:
>> >>>
>> >>>
>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>>>
>> >>>> http://NNHOST:50070/conf
>> >>>
>> >>>
>> >>>
>> >>> Harsh, I changed the value as said & restarted service NN. For
>> verifying
>> >>> i checked the http link that you gave and i saw the property their
>> but on
>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
>> >>> blocks. Please check the logs or run fsck in order to identify the
>> missing
>> >>> blocks.) when i clicked on this link i can see file names . Do I
>> need to
>> >>> reboot the machine to run fsck on root fs/ or is there hadoop command
>> fsck
>> >>> that i can run on the running hadoop ?
>> >>>
>> >>> Thanks,
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
After creating the directory and setting permission I tried to restart the
services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are not
being started.
Need to check few logs from below logs.
===================================
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
************************************************************/
2013-02-28 05:06:23,385 WARN org.apache.hadoop.metrics2.impl.MetricsConfig:
Cannot locate configuration: tried
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
started
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2013-02-28 05:06:25,623 INFO
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=1000
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.block.access.token.enable=false
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
defaultReplication = 1
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxReplicationStreams = 2
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
shouldCheckForEnoughRacks = false
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
replicationRecheckInterval = 3000
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
encryptDataTransfer = false
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
hdfs (auth:SIMPLE)
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
hadmin
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true
2013-02-28 05:06:26,023 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 05:06:26,026 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 05:06:26,359 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.extension = 0
2013-02-28 05:06:26,378 INFO org.apache.hadoop.hdfs.server.common.Storage:
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
7275@OPERA-MAST1.ny.os.local
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:06:26,382 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2013-02-28 05:06:26,394 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
> If I do compare with the older path the permissions are same on the parent
> directories.
> Do I need to create this this directory manually and set the permission ?
>
> Older Path
>
> # ll /var/lib/hadoop-hdfs/cache/hdfs/
> total 4
> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>
> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
> total 12
> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>
>
> New Path
>
> # ll /mnt/san1/hdfs/cache/hdfs/
> total 4
> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>
>
> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
> total 4
> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>
>
> Thanks,
>
>
>
> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi,
>>
>> The exact error is displayed on your log and should be somewhat self
>> explanatory:
>>
>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>> state: storage directory does not exist or is not accessible.
>>
>> Please check this one's availability, permissions (the NN user should
>> be able to access it).
>>
>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>> <pr...@gmail.com> wrote:
>> > Please find below logs for shutting down the namenode service. Can
>> anybody
>> > check this
>> >
>> > 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path
>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> > configuration files. Please update hdfs configuration.
>> > 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path
>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> > configuration files. Please update hdfs configuration.
>> > 2013-02-28 02:07:51,754 WARN
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>> storage
>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack
>> > of redundant storage directories!
>> > 2013-02-28 02:07:51,754 WARN
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> > dataloss due to lack of redundant storage directories!
>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>> > Refreshing hosts (include/exclude) list
>> > 2013-02-28 02:07:51,890 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> > dfs.block.invalidate.limit=1000
>> > 2013-02-28 02:07:51,909 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > dfs.block.access.token.enable=false
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > defaultReplication = 1
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplication
>> > = 512
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> minReplication
>> > = 1
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > maxReplicationStreams = 2
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > shouldCheckForEnoughRacks = false
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > replicationRecheckInterval = 3000
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > encryptDataTransfer = false
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>> =
>> > hdfs (auth:SIMPLE)
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>> =
>> > hadmin
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> isPermissionEnabled =
>> > true
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>> > 2013-02-28 02:07:51,925 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>> true
>> > 2013-02-28 02:07:52,462 INFO
>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring
>> > more than 10 times
>> > 2013-02-28 02:07:52,466 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> > 2013-02-28 02:07:52,467 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> > dfs.namenode.safemode.min.datanodes = 0
>> > 2013-02-28 02:07:52,467 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> > dfs.namenode.safemode.extension = 0
>> > 2013-02-28 02:07:52,469 INFO
>> org.apache.hadoop.hdfs.server.common.Storage:
>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>> > 2013-02-28 02:07:52,471 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics
>> > system...
>> > 2013-02-28 02:07:52,472 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> > stopped.
>> > 2013-02-28 02:07:52,473 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> > shutdown complete.
>> > 2013-02-28 02:07:52,473 FATAL
>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
>> join
>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> Directory
>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
>> > directory does not exist or is not accessible.
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> > at
>> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> with
>> > status 1
>> > 2013-02-28 02:08:48,913 INFO
>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> > /************************************************************
>> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
>> 192.168.1.3
>> >
>> >
>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>> > <pr...@gmail.com> wrote:
>> >>
>> >> Hi Guys,
>> >>
>> >> I have space on other partition. Can I change the path for cache files
>> on
>> >> other partition ? I have below properties . Can it resolve the issue ?
>> If i
>> >> change the path to other directories and restart services I get the
>> below
>> >> error while starting the service namenode. I didn't find anything in
>> logs so
>> >> far. Can you please suggest something ?
>> >>
>> >> <property>
>> >> <name>hadoop.tmp.dir</name>
>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>> >> </property>
>> >> <property>
>> >> <name>dfs.namenode.name.dir</name>
>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>> >> </property>
>> >> <property>
>> >> <name>dfs.namenode.checkpoint.dir</name>
>> >>
>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/namesecondary</value>
>> >> </property>
>> >> <property>
>> >>
>> >>
>> >> Service namenode is failing
>> >>
>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
>> done
>> >> Hadoop datanode is running [ OK ]
>> >> Hadoop namenode is dead and pid file exists [FAILED]
>> >> Hadoop secondarynamenode is running [ OK ]
>> >>
>> >> Thanks,
>> >>
>> >>
>> >>
>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>> >> <pr...@gmail.com> wrote:
>> >>>
>> >>>
>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>>>
>> >>>> http://NNHOST:50070/conf
>> >>>
>> >>>
>> >>>
>> >>> Harsh, I changed the value as said & restarted service NN. For
>> verifying
>> >>> i checked the http link that you gave and i saw the property their
>> but on
>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
>> >>> blocks. Please check the logs or run fsck in order to identify the
>> missing
>> >>> blocks.) when i clicked on this link i can see file names . Do I
>> need to
>> >>> reboot the machine to run fsck on root fs/ or is there hadoop command
>> fsck
>> >>> that i can run on the running hadoop ?
>> >>>
>> >>> Thanks,
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
After creating the directory and setting permission I tried to restart the
services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are not
being started.
Need to check few logs from below logs.
===================================
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
************************************************************/
2013-02-28 05:06:23,385 WARN org.apache.hadoop.metrics2.impl.MetricsConfig:
Cannot locate configuration: tried
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
started
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2013-02-28 05:06:25,623 INFO
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=1000
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.block.access.token.enable=false
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
defaultReplication = 1
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxReplicationStreams = 2
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
shouldCheckForEnoughRacks = false
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
replicationRecheckInterval = 3000
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
encryptDataTransfer = false
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
hdfs (auth:SIMPLE)
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
hadmin
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true
2013-02-28 05:06:26,023 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 05:06:26,026 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 05:06:26,359 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.extension = 0
2013-02-28 05:06:26,378 INFO org.apache.hadoop.hdfs.server.common.Storage:
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
7275@OPERA-MAST1.ny.os.local
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:06:26,382 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2013-02-28 05:06:26,394 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
> If I do compare with the older path the permissions are same on the parent
> directories.
> Do I need to create this this directory manually and set the permission ?
>
> Older Path
>
> # ll /var/lib/hadoop-hdfs/cache/hdfs/
> total 4
> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>
> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
> total 12
> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>
>
> New Path
>
> # ll /mnt/san1/hdfs/cache/hdfs/
> total 4
> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>
>
> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
> total 4
> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>
>
> Thanks,
>
>
>
> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi,
>>
>> The exact error is displayed on your log and should be somewhat self
>> explanatory:
>>
>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>> state: storage directory does not exist or is not accessible.
>>
>> Please check this one's availability, permissions (the NN user should
>> be able to access it).
>>
>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>> <pr...@gmail.com> wrote:
>> > Please find below logs for shutting down the namenode service. Can
>> anybody
>> > check this
>> >
>> > 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path
>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> > configuration files. Please update hdfs configuration.
>> > 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path
>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> > configuration files. Please update hdfs configuration.
>> > 2013-02-28 02:07:51,754 WARN
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>> storage
>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack
>> > of redundant storage directories!
>> > 2013-02-28 02:07:51,754 WARN
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> > dataloss due to lack of redundant storage directories!
>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>> > Refreshing hosts (include/exclude) list
>> > 2013-02-28 02:07:51,890 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> > dfs.block.invalidate.limit=1000
>> > 2013-02-28 02:07:51,909 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > dfs.block.access.token.enable=false
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > defaultReplication = 1
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplication
>> > = 512
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> minReplication
>> > = 1
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > maxReplicationStreams = 2
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > shouldCheckForEnoughRacks = false
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > replicationRecheckInterval = 3000
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > encryptDataTransfer = false
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>> =
>> > hdfs (auth:SIMPLE)
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>> =
>> > hadmin
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> isPermissionEnabled =
>> > true
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>> > 2013-02-28 02:07:51,925 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>> true
>> > 2013-02-28 02:07:52,462 INFO
>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring
>> > more than 10 times
>> > 2013-02-28 02:07:52,466 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> > 2013-02-28 02:07:52,467 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> > dfs.namenode.safemode.min.datanodes = 0
>> > 2013-02-28 02:07:52,467 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> > dfs.namenode.safemode.extension = 0
>> > 2013-02-28 02:07:52,469 INFO
>> org.apache.hadoop.hdfs.server.common.Storage:
>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>> > 2013-02-28 02:07:52,471 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics
>> > system...
>> > 2013-02-28 02:07:52,472 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> > stopped.
>> > 2013-02-28 02:07:52,473 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> > shutdown complete.
>> > 2013-02-28 02:07:52,473 FATAL
>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
>> join
>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> Directory
>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
>> > directory does not exist or is not accessible.
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> > at
>> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> with
>> > status 1
>> > 2013-02-28 02:08:48,913 INFO
>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> > /************************************************************
>> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
>> 192.168.1.3
>> >
>> >
>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>> > <pr...@gmail.com> wrote:
>> >>
>> >> Hi Guys,
>> >>
>> >> I have space on other partition. Can I change the path for cache files
>> on
>> >> other partition ? I have below properties . Can it resolve the issue ?
>> If i
>> >> change the path to other directories and restart services I get the
>> below
>> >> error while starting the service namenode. I didn't find anything in
>> logs so
>> >> far. Can you please suggest something ?
>> >>
>> >> <property>
>> >> <name>hadoop.tmp.dir</name>
>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>> >> </property>
>> >> <property>
>> >> <name>dfs.namenode.name.dir</name>
>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>> >> </property>
>> >> <property>
>> >> <name>dfs.namenode.checkpoint.dir</name>
>> >>
>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/namesecondary</value>
>> >> </property>
>> >> <property>
>> >>
>> >>
>> >> Service namenode is failing
>> >>
>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
>> done
>> >> Hadoop datanode is running [ OK ]
>> >> Hadoop namenode is dead and pid file exists [FAILED]
>> >> Hadoop secondarynamenode is running [ OK ]
>> >>
>> >> Thanks,
>> >>
>> >>
>> >>
>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>> >> <pr...@gmail.com> wrote:
>> >>>
>> >>>
>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>>>
>> >>>> http://NNHOST:50070/conf
>> >>>
>> >>>
>> >>>
>> >>> Harsh, I changed the value as said & restarted service NN. For
>> verifying
>> >>> i checked the http link that you gave and i saw the property their
>> but on
>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
>> >>> blocks. Please check the logs or run fsck in order to identify the
>> missing
>> >>> blocks.) when i clicked on this link i can see file names . Do I
>> need to
>> >>> reboot the machine to run fsck on root fs/ or is there hadoop command
>> fsck
>> >>> that i can run on the running hadoop ?
>> >>>
>> >>> Thanks,
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
After creating the directory and setting permission I tried to restart the
services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
acquired by nodename 7275@OPERA-MAST1.ny.os.local" and services are not
being started.
Need to check few logs from below logs.
===================================
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
************************************************************/
2013-02-28 05:06:23,385 WARN org.apache.hadoop.metrics2.impl.MetricsConfig:
Cannot locate configuration: tried
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
started
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2013-02-28 05:06:25,623 INFO
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=1000
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.block.access.token.enable=false
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
defaultReplication = 1
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxReplicationStreams = 2
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
shouldCheckForEnoughRacks = false
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
replicationRecheckInterval = 3000
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
encryptDataTransfer = false
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
hdfs (auth:SIMPLE)
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
hadmin
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true
2013-02-28 05:06:26,023 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 05:06:26,026 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 05:06:26,359 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.extension = 0
2013-02-28 05:06:26,378 INFO org.apache.hadoop.hdfs.server.common.Storage:
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
7275@OPERA-MAST1.ny.os.local
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:06:26,382 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2013-02-28 05:06:26,394 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera <project.linux.proj@gmail.com
> wrote:
> Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
> If I do compare with the older path the permissions are same on the parent
> directories.
> Do I need to create this this directory manually and set the permission ?
>
> Older Path
>
> # ll /var/lib/hadoop-hdfs/cache/hdfs/
> total 4
> drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
>
> # ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
> total 12
> drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
> drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>
>
> New Path
>
> # ll /mnt/san1/hdfs/cache/hdfs/
> total 4
> drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
>
>
> # ll /mnt/san1/hdfs/cache/hdfs/dfs/
> total 4
> drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
>
>
> Thanks,
>
>
>
> On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi,
>>
>> The exact error is displayed on your log and should be somewhat self
>> explanatory:
>>
>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
>> state: storage directory does not exist or is not accessible.
>>
>> Please check this one's availability, permissions (the NN user should
>> be able to access it).
>>
>> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
>> <pr...@gmail.com> wrote:
>> > Please find below logs for shutting down the namenode service. Can
>> anybody
>> > check this
>> >
>> > 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path
>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> > configuration files. Please update hdfs configuration.
>> > 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
>> Path
>> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
>> > configuration files. Please update hdfs configuration.
>> > 2013-02-28 02:07:51,754 WARN
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
>> storage
>> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack
>> > of redundant storage directories!
>> > 2013-02-28 02:07:51,754 WARN
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> > dataloss due to lack of redundant storage directories!
>> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
>> > Refreshing hosts (include/exclude) list
>> > 2013-02-28 02:07:51,890 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> > dfs.block.invalidate.limit=1000
>> > 2013-02-28 02:07:51,909 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > dfs.block.access.token.enable=false
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > defaultReplication = 1
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplication
>> > = 512
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> minReplication
>> > = 1
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > maxReplicationStreams = 2
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > shouldCheckForEnoughRacks = false
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > replicationRecheckInterval = 3000
>> > 2013-02-28 02:07:51,910 INFO
>> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> > encryptDataTransfer = false
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
>> =
>> > hdfs (auth:SIMPLE)
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
>> =
>> > hadmin
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> isPermissionEnabled =
>> > true
>> > 2013-02-28 02:07:51,920 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>> > 2013-02-28 02:07:51,925 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
>> true
>> > 2013-02-28 02:07:52,462 INFO
>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring
>> > more than 10 times
>> > 2013-02-28 02:07:52,466 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>> > 2013-02-28 02:07:52,467 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> > dfs.namenode.safemode.min.datanodes = 0
>> > 2013-02-28 02:07:52,467 INFO
>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> > dfs.namenode.safemode.extension = 0
>> > 2013-02-28 02:07:52,469 INFO
>> org.apache.hadoop.hdfs.server.common.Storage:
>> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
>> > 2013-02-28 02:07:52,471 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics
>> > system...
>> > 2013-02-28 02:07:52,472 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> > stopped.
>> > 2013-02-28 02:07:52,473 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>> system
>> > shutdown complete.
>> > 2013-02-28 02:07:52,473 FATAL
>> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
>> join
>> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> Directory
>> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
>> > directory does not exist or is not accessible.
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>> > at
>> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
>> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> with
>> > status 1
>> > 2013-02-28 02:08:48,913 INFO
>> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> > /************************************************************
>> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
>> 192.168.1.3
>> >
>> >
>> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
>> > <pr...@gmail.com> wrote:
>> >>
>> >> Hi Guys,
>> >>
>> >> I have space on other partition. Can I change the path for cache files
>> on
>> >> other partition ? I have below properties . Can it resolve the issue ?
>> If i
>> >> change the path to other directories and restart services I get the
>> below
>> >> error while starting the service namenode. I didn't find anything in
>> logs so
>> >> far. Can you please suggest something ?
>> >>
>> >> <property>
>> >> <name>hadoop.tmp.dir</name>
>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>> >> </property>
>> >> <property>
>> >> <name>dfs.namenode.name.dir</name>
>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>> >> </property>
>> >> <property>
>> >> <name>dfs.namenode.checkpoint.dir</name>
>> >>
>> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
>> }/dfs/namesecondary</value>
>> >> </property>
>> >> <property>
>> >>
>> >>
>> >> Service namenode is failing
>> >>
>> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
>> done
>> >> Hadoop datanode is running [ OK ]
>> >> Hadoop namenode is dead and pid file exists [FAILED]
>> >> Hadoop secondarynamenode is running [ OK ]
>> >>
>> >> Thanks,
>> >>
>> >>
>> >>
>> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>> >> <pr...@gmail.com> wrote:
>> >>>
>> >>>
>> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>>>
>> >>>> http://NNHOST:50070/conf
>> >>>
>> >>>
>> >>>
>> >>> Harsh, I changed the value as said & restarted service NN. For
>> verifying
>> >>> i checked the http link that you gave and i saw the property their
>> but on
>> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
>> >>> blocks. Please check the logs or run fsck in order to identify the
>> missing
>> >>> blocks.) when i clicked on this link i can see file names . Do I
>> need to
>> >>> reboot the machine to run fsck on root fs/ or is there hadoop command
>> fsck
>> >>> that i can run on the running hadoop ?
>> >>>
>> >>> Thanks,
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created . If
I do compare with the older path the permissions are same on the parent
directories.
Do I need to create this this directory manually and set the permission ?
Older Path
# ll /var/lib/hadoop-hdfs/cache/hdfs/
total 4
drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
# ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
New Path
# ll /mnt/san1/hdfs/cache/hdfs/
total 4
drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
# ll /mnt/san1/hdfs/cache/hdfs/dfs/
total 4
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
Thanks,
On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> The exact error is displayed on your log and should be somewhat self
> explanatory:
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
> state: storage directory does not exist or is not accessible.
>
> Please check this one's availability, permissions (the NN user should
> be able to access it).
>
> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
> <pr...@gmail.com> wrote:
> > Please find below logs for shutting down the namenode service. Can
> anybody
> > check this
> >
> > 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path
> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> > configuration files. Please update hdfs configuration.
> > 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path
> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> > configuration files. Please update hdfs configuration.
> > 2013-02-28 02:07:51,754 WARN
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> storage
> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack
> > of redundant storage directories!
> > 2013-02-28 02:07:51,754 WARN
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> > dataloss due to lack of redundant storage directories!
> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
> > Refreshing hosts (include/exclude) list
> > 2013-02-28 02:07:51,890 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> > dfs.block.invalidate.limit=1000
> > 2013-02-28 02:07:51,909 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > dfs.block.access.token.enable=false
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > defaultReplication = 1
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication
> > = 512
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication
> > = 1
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > maxReplicationStreams = 2
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > shouldCheckForEnoughRacks = false
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > replicationRecheckInterval = 3000
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > encryptDataTransfer = false
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
> =
> > hdfs (auth:SIMPLE)
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
> =
> > hadmin
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled
> =
> > true
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> > 2013-02-28 02:07:51,925 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> > 2013-02-28 02:07:52,462 INFO
> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring
> > more than 10 times
> > 2013-02-28 02:07:52,466 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> > 2013-02-28 02:07:52,467 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> > dfs.namenode.safemode.min.datanodes = 0
> > 2013-02-28 02:07:52,467 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> > dfs.namenode.safemode.extension = 0
> > 2013-02-28 02:07:52,469 INFO
> org.apache.hadoop.hdfs.server.common.Storage:
> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
> > 2013-02-28 02:07:52,471 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics
> > system...
> > 2013-02-28 02:07:52,472 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> > stopped.
> > 2013-02-28 02:07:52,473 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> > shutdown complete.
> > 2013-02-28 02:07:52,473 FATAL
> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
> join
> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory
> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
> > directory does not exist or is not accessible.
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
> with
> > status 1
> > 2013-02-28 02:08:48,913 INFO
> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> > /************************************************************
> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
> 192.168.1.3
> >
> >
> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
> > <pr...@gmail.com> wrote:
> >>
> >> Hi Guys,
> >>
> >> I have space on other partition. Can I change the path for cache files
> on
> >> other partition ? I have below properties . Can it resolve the issue ?
> If i
> >> change the path to other directories and restart services I get the
> below
> >> error while starting the service namenode. I didn't find anything in
> logs so
> >> far. Can you please suggest something ?
> >>
> >> <property>
> >> <name>hadoop.tmp.dir</name>
> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> >> </property>
> >> <property>
> >> <name>dfs.namenode.name.dir</name>
> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
> >> </property>
> >> <property>
> >> <name>dfs.namenode.checkpoint.dir</name>
> >>
> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/namesecondary</value>
> >> </property>
> >> <property>
> >>
> >>
> >> Service namenode is failing
> >>
> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
> done
> >> Hadoop datanode is running [ OK ]
> >> Hadoop namenode is dead and pid file exists [FAILED]
> >> Hadoop secondarynamenode is running [ OK ]
> >>
> >> Thanks,
> >>
> >>
> >>
> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
> >> <pr...@gmail.com> wrote:
> >>>
> >>>
> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com> wrote:
> >>>>
> >>>> http://NNHOST:50070/conf
> >>>
> >>>
> >>>
> >>> Harsh, I changed the value as said & restarted service NN. For
> verifying
> >>> i checked the http link that you gave and i saw the property their but
> on
> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
> >>> blocks. Please check the logs or run fsck in order to identify the
> missing
> >>> blocks.) when i clicked on this link i can see file names . Do I
> need to
> >>> reboot the machine to run fsck on root fs/ or is there hadoop command
> fsck
> >>> that i can run on the running hadoop ?
> >>>
> >>> Thanks,
> >>>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created . If
I do compare with the older path the permissions are same on the parent
directories.
Do I need to create this this directory manually and set the permission ?
Older Path
# ll /var/lib/hadoop-hdfs/cache/hdfs/
total 4
drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
# ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
New Path
# ll /mnt/san1/hdfs/cache/hdfs/
total 4
drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
# ll /mnt/san1/hdfs/cache/hdfs/dfs/
total 4
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
Thanks,
On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> The exact error is displayed on your log and should be somewhat self
> explanatory:
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
> state: storage directory does not exist or is not accessible.
>
> Please check this one's availability, permissions (the NN user should
> be able to access it).
>
> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
> <pr...@gmail.com> wrote:
> > Please find below logs for shutting down the namenode service. Can
> anybody
> > check this
> >
> > 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path
> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> > configuration files. Please update hdfs configuration.
> > 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path
> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> > configuration files. Please update hdfs configuration.
> > 2013-02-28 02:07:51,754 WARN
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> storage
> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack
> > of redundant storage directories!
> > 2013-02-28 02:07:51,754 WARN
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> > dataloss due to lack of redundant storage directories!
> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
> > Refreshing hosts (include/exclude) list
> > 2013-02-28 02:07:51,890 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> > dfs.block.invalidate.limit=1000
> > 2013-02-28 02:07:51,909 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > dfs.block.access.token.enable=false
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > defaultReplication = 1
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication
> > = 512
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication
> > = 1
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > maxReplicationStreams = 2
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > shouldCheckForEnoughRacks = false
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > replicationRecheckInterval = 3000
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > encryptDataTransfer = false
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
> =
> > hdfs (auth:SIMPLE)
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
> =
> > hadmin
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled
> =
> > true
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> > 2013-02-28 02:07:51,925 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> > 2013-02-28 02:07:52,462 INFO
> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring
> > more than 10 times
> > 2013-02-28 02:07:52,466 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> > 2013-02-28 02:07:52,467 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> > dfs.namenode.safemode.min.datanodes = 0
> > 2013-02-28 02:07:52,467 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> > dfs.namenode.safemode.extension = 0
> > 2013-02-28 02:07:52,469 INFO
> org.apache.hadoop.hdfs.server.common.Storage:
> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
> > 2013-02-28 02:07:52,471 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics
> > system...
> > 2013-02-28 02:07:52,472 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> > stopped.
> > 2013-02-28 02:07:52,473 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> > shutdown complete.
> > 2013-02-28 02:07:52,473 FATAL
> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
> join
> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory
> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
> > directory does not exist or is not accessible.
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
> with
> > status 1
> > 2013-02-28 02:08:48,913 INFO
> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> > /************************************************************
> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
> 192.168.1.3
> >
> >
> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
> > <pr...@gmail.com> wrote:
> >>
> >> Hi Guys,
> >>
> >> I have space on other partition. Can I change the path for cache files
> on
> >> other partition ? I have below properties . Can it resolve the issue ?
> If i
> >> change the path to other directories and restart services I get the
> below
> >> error while starting the service namenode. I didn't find anything in
> logs so
> >> far. Can you please suggest something ?
> >>
> >> <property>
> >> <name>hadoop.tmp.dir</name>
> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> >> </property>
> >> <property>
> >> <name>dfs.namenode.name.dir</name>
> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
> >> </property>
> >> <property>
> >> <name>dfs.namenode.checkpoint.dir</name>
> >>
> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/namesecondary</value>
> >> </property>
> >> <property>
> >>
> >>
> >> Service namenode is failing
> >>
> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
> done
> >> Hadoop datanode is running [ OK ]
> >> Hadoop namenode is dead and pid file exists [FAILED]
> >> Hadoop secondarynamenode is running [ OK ]
> >>
> >> Thanks,
> >>
> >>
> >>
> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
> >> <pr...@gmail.com> wrote:
> >>>
> >>>
> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com> wrote:
> >>>>
> >>>> http://NNHOST:50070/conf
> >>>
> >>>
> >>>
> >>> Harsh, I changed the value as said & restarted service NN. For
> verifying
> >>> i checked the http link that you gave and i saw the property their but
> on
> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
> >>> blocks. Please check the logs or run fsck in order to identify the
> missing
> >>> blocks.) when i clicked on this link i can see file names . Do I
> need to
> >>> reboot the machine to run fsck on root fs/ or is there hadoop command
> fsck
> >>> that i can run on the running hadoop ?
> >>>
> >>> Thanks,
> >>>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created . If
I do compare with the older path the permissions are same on the parent
directories.
Do I need to create this this directory manually and set the permission ?
Older Path
# ll /var/lib/hadoop-hdfs/cache/hdfs/
total 4
drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
# ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
New Path
# ll /mnt/san1/hdfs/cache/hdfs/
total 4
drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
# ll /mnt/san1/hdfs/cache/hdfs/dfs/
total 4
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
Thanks,
On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> The exact error is displayed on your log and should be somewhat self
> explanatory:
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
> state: storage directory does not exist or is not accessible.
>
> Please check this one's availability, permissions (the NN user should
> be able to access it).
>
> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
> <pr...@gmail.com> wrote:
> > Please find below logs for shutting down the namenode service. Can
> anybody
> > check this
> >
> > 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path
> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> > configuration files. Please update hdfs configuration.
> > 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path
> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> > configuration files. Please update hdfs configuration.
> > 2013-02-28 02:07:51,754 WARN
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> storage
> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack
> > of redundant storage directories!
> > 2013-02-28 02:07:51,754 WARN
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> > dataloss due to lack of redundant storage directories!
> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
> > Refreshing hosts (include/exclude) list
> > 2013-02-28 02:07:51,890 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> > dfs.block.invalidate.limit=1000
> > 2013-02-28 02:07:51,909 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > dfs.block.access.token.enable=false
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > defaultReplication = 1
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication
> > = 512
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication
> > = 1
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > maxReplicationStreams = 2
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > shouldCheckForEnoughRacks = false
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > replicationRecheckInterval = 3000
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > encryptDataTransfer = false
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
> =
> > hdfs (auth:SIMPLE)
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
> =
> > hadmin
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled
> =
> > true
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> > 2013-02-28 02:07:51,925 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> > 2013-02-28 02:07:52,462 INFO
> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring
> > more than 10 times
> > 2013-02-28 02:07:52,466 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> > 2013-02-28 02:07:52,467 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> > dfs.namenode.safemode.min.datanodes = 0
> > 2013-02-28 02:07:52,467 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> > dfs.namenode.safemode.extension = 0
> > 2013-02-28 02:07:52,469 INFO
> org.apache.hadoop.hdfs.server.common.Storage:
> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
> > 2013-02-28 02:07:52,471 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics
> > system...
> > 2013-02-28 02:07:52,472 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> > stopped.
> > 2013-02-28 02:07:52,473 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> > shutdown complete.
> > 2013-02-28 02:07:52,473 FATAL
> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
> join
> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory
> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
> > directory does not exist or is not accessible.
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
> with
> > status 1
> > 2013-02-28 02:08:48,913 INFO
> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> > /************************************************************
> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
> 192.168.1.3
> >
> >
> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
> > <pr...@gmail.com> wrote:
> >>
> >> Hi Guys,
> >>
> >> I have space on other partition. Can I change the path for cache files
> on
> >> other partition ? I have below properties . Can it resolve the issue ?
> If i
> >> change the path to other directories and restart services I get the
> below
> >> error while starting the service namenode. I didn't find anything in
> logs so
> >> far. Can you please suggest something ?
> >>
> >> <property>
> >> <name>hadoop.tmp.dir</name>
> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> >> </property>
> >> <property>
> >> <name>dfs.namenode.name.dir</name>
> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
> >> </property>
> >> <property>
> >> <name>dfs.namenode.checkpoint.dir</name>
> >>
> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/namesecondary</value>
> >> </property>
> >> <property>
> >>
> >>
> >> Service namenode is failing
> >>
> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
> done
> >> Hadoop datanode is running [ OK ]
> >> Hadoop namenode is dead and pid file exists [FAILED]
> >> Hadoop secondarynamenode is running [ OK ]
> >>
> >> Thanks,
> >>
> >>
> >>
> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
> >> <pr...@gmail.com> wrote:
> >>>
> >>>
> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com> wrote:
> >>>>
> >>>> http://NNHOST:50070/conf
> >>>
> >>>
> >>>
> >>> Harsh, I changed the value as said & restarted service NN. For
> verifying
> >>> i checked the http link that you gave and i saw the property their but
> on
> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
> >>> blocks. Please check the logs or run fsck in order to identify the
> missing
> >>> blocks.) when i clicked on this link i can see file names . Do I
> need to
> >>> reboot the machine to run fsck on root fs/ or is there hadoop command
> fsck
> >>> that i can run on the running hadoop ?
> >>>
> >>> Thanks,
> >>>
> >>
> >
>
>
>
> --
> Harsh J
>
Re: NameNode low on available disk space
Posted by Mohit Vadhera <pr...@gmail.com>.
Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created . If
I do compare with the older path the permissions are same on the parent
directories.
Do I need to create this this directory manually and set the permission ?
Older Path
# ll /var/lib/hadoop-hdfs/cache/hdfs/
total 4
drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
# ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
New Path
# ll /mnt/san1/hdfs/cache/hdfs/
total 4
drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
# ll /mnt/san1/hdfs/cache/hdfs/dfs/
total 4
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
Thanks,
On Thu, Feb 28, 2013 at 1:59 PM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> The exact error is displayed on your log and should be somewhat self
> explanatory:
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
> state: storage directory does not exist or is not accessible.
>
> Please check this one's availability, permissions (the NN user should
> be able to access it).
>
> On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
> <pr...@gmail.com> wrote:
> > Please find below logs for shutting down the namenode service. Can
> anybody
> > check this
> >
> > 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path
> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> > configuration files. Please update hdfs configuration.
> > 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util:
> Path
> > /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> > configuration files. Please update hdfs configuration.
> > 2013-02-28 02:07:51,754 WARN
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
> storage
> > directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack
> > of redundant storage directories!
> > 2013-02-28 02:07:51,754 WARN
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> > edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> > dataloss due to lack of redundant storage directories!
> > 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
> > Refreshing hosts (include/exclude) list
> > 2013-02-28 02:07:51,890 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> > dfs.block.invalidate.limit=1000
> > 2013-02-28 02:07:51,909 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > dfs.block.access.token.enable=false
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > defaultReplication = 1
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication
> > = 512
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication
> > = 1
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > maxReplicationStreams = 2
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > shouldCheckForEnoughRacks = false
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > replicationRecheckInterval = 3000
> > 2013-02-28 02:07:51,910 INFO
> > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> > encryptDataTransfer = false
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
> =
> > hdfs (auth:SIMPLE)
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
> =
> > hadmin
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled
> =
> > true
> > 2013-02-28 02:07:51,920 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> > 2013-02-28 02:07:51,925 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> > 2013-02-28 02:07:52,462 INFO
> > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring
> > more than 10 times
> > 2013-02-28 02:07:52,466 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> > dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> > 2013-02-28 02:07:52,467 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> > dfs.namenode.safemode.min.datanodes = 0
> > 2013-02-28 02:07:52,467 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> > dfs.namenode.safemode.extension = 0
> > 2013-02-28 02:07:52,469 INFO
> org.apache.hadoop.hdfs.server.common.Storage:
> > Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
> > 2013-02-28 02:07:52,471 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics
> > system...
> > 2013-02-28 02:07:52,472 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> > stopped.
> > 2013-02-28 02:07:52,473 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system
> > shutdown complete.
> > 2013-02-28 02:07:52,473 FATAL
> > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
> join
> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory
> > /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
> > directory does not exist or is not accessible.
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> > 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
> with
> > status 1
> > 2013-02-28 02:08:48,913 INFO
> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> > /************************************************************
> > SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
> 192.168.1.3
> >
> >
> > On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
> > <pr...@gmail.com> wrote:
> >>
> >> Hi Guys,
> >>
> >> I have space on other partition. Can I change the path for cache files
> on
> >> other partition ? I have below properties . Can it resolve the issue ?
> If i
> >> change the path to other directories and restart services I get the
> below
> >> error while starting the service namenode. I didn't find anything in
> logs so
> >> far. Can you please suggest something ?
> >>
> >> <property>
> >> <name>hadoop.tmp.dir</name>
> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
> >> </property>
> >> <property>
> >> <name>dfs.namenode.name.dir</name>
> >> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
> >> </property>
> >> <property>
> >> <name>dfs.namenode.checkpoint.dir</name>
> >>
> >> <value>/var/lib/hadoop-hdfs/cache/${user.name
> }/dfs/namesecondary</value>
> >> </property>
> >> <property>
> >>
> >>
> >> Service namenode is failing
> >>
> >> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
> done
> >> Hadoop datanode is running [ OK ]
> >> Hadoop namenode is dead and pid file exists [FAILED]
> >> Hadoop secondarynamenode is running [ OK ]
> >>
> >> Thanks,
> >>
> >>
> >>
> >> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
> >> <pr...@gmail.com> wrote:
> >>>
> >>>
> >>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <ha...@cloudera.com> wrote:
> >>>>
> >>>> http://NNHOST:50070/conf
> >>>
> >>>
> >>>
> >>> Harsh, I changed the value as said & restarted service NN. For
> verifying
> >>> i checked the http link that you gave and i saw the property their but
> on
> >>> http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
> >>> blocks. Please check the logs or run fsck in order to identify the
> missing
> >>> blocks.) when i clicked on this link i can see file names . Do I
> need to
> >>> reboot the machine to run fsck on root fs/ or is there hadoop command
> fsck
> >>> that i can run on the running hadoop ?
> >>>
> >>> Thanks,
> >>>
> >>
> >
>
>
>
> --
> Harsh J
>