You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-user@hadoop.apache.org by Krishna Kishore Bonagiri <wr...@gmail.com> on 2013/07/15 08:44:43 UTC

Namenode automatically going to safemode with 2.1.0-beta

Hi,

 I am doing no activity on my single node cluster which is using
2.1.0-beta, and still observed that it has gone to safe mode by itself
after a while. I was looking at the name node log and see many of these
kinds of entries.. Can anything be interpreted from these?

2013-07-12 09:06:11,256 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
561
2013-07-12 09:07:11,290 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
9.70.137.114
2013-07-12 09:07:11,290 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
2013-07-12 09:07:11,290 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
2013-07-12 09:07:11,291 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
Total time for transactions(ms): 1 Number of transactions batched in Syncs:
0 Number of syncs: 2 SyncTimes(ms): 14
2013-07-12 09:07:11,292 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
Total time for transactions(ms): 1 Number of transactions batched in Syncs:
0 Number of syncs: 3 SyncTimes(ms): 15
2013-07-12 09:07:11,293 INFO
org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
file
/tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
/tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
2013-07-12 09:07:11,294 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
563
2013-07-12 09:08:11,397 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
9.70.137.114
2013-07-12 09:08:11,398 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
2013-07-12 09:08:11,398 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
2013-07-12 09:08:11,399 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
Total time for transactions(ms): 2 Number of transactions batched in Syncs:
0 Number of syncs: 2 SyncTimes(ms): 11
2013-07-12 09:08:11,400 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
Total time for transactions(ms): 2 Number of transactions batched in Syncs:
0 Number of syncs: 3 SyncTimes(ms): 12
2013-07-12 09:08:11,402 INFO
org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
file
/tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
/tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
2013-07-12 09:08:11,402 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
565
2013-07-12 09:09:11,440 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
9.70.137.114
2013-07-12 09:09:11,440 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
2013-07-12 09:09:11,440 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
2013-07-12 09:09:11,440 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
Total time for transactions(ms): 0 Number of transactions batched in Syncs:
0 Number of syncs: 2 SyncTimes(ms): 13
2013-07-12 09:09:11,441 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
Total time for transactions(ms): 0 Number of transactions batched in Syncs:
0 Number of syncs: 3 SyncTimes(ms): 13


And after sometime it said:

2013-07-12 11:03:19,799 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
795
2013-07-12 11:04:19,826 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
9.70.137.114
2013-07-12 11:04:19,826 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
2013-07-12 11:04:19,827 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
2013-07-12 11:04:19,827 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
Total time for transactions(ms): 0 Number of transactions batched in Syncs:
0 Number of syncs: 2 SyncTimes(ms): 12
2013-07-12 11:04:19,827 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
Total time for transactions(ms): 0 Number of transactions batched in Syncs:
0 Number of syncs: 3 SyncTimes(ms): 12
2013-07-12 11:04:19,829 INFO
org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
file
/tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
/tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
2013-07-12 11:04:19,829 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
797
2013-07-12 11:04:26,002 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume 'null' is 0, which is below the configured reserved
amount 104857600
2013-07-12 11:04:26,003 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
available disk space. Entering safe mode.
2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Safe mode is ON.
Resources are low on NN. Please add or free up more resources then turn off
safe mode manually.  NOTE:  If you turn off safe mode before adding
resources, the NN will immediately return to safe mode..
2013-07-12 11:04:31,004 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume 'null' is 0, which is below the configured reserved
amount 104857600
2013-07-12 11:04:31,005 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
available disk space. Already in safe mode.
2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Safe mode is ONResources are low on NN. Please add or free up more
resources then turn off safe mode manually.  NOTE:  If you turn off safe
mode before adding resources, the NN will immediately return to safe mode..
2013-07-12 11:04:36,005 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume 'null' is 0, which is below the configured reserved
amount 104857600


Thanks,
Kishore

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Azuryy Yu <az...@gmail.com>.

hi,
from the log:
NameNode low on available disk space. Entering safe mode.

this is the root cause.
 On Jul 15, 2013 2:45 PM, "Krishna Kishore Bonagiri" <
write2kishore@gmail.com> wrote:

> Hi,
>
>  I am doing no activity on my single node cluster which is using
> 2.1.0-beta, and still observed that it has gone to safe mode by itself
> after a while. I was looking at the name node log and see many of these
> kinds of entries.. Can anything be interpreted from these?
>
> 2013-07-12 09:06:11,256 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 561
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
> 2013-07-12 09:07:11,291 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 14
> 2013-07-12 09:07:11,292 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 15
> 2013-07-12 09:07:11,293 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> 2013-07-12 09:07:11,294 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 563
> 2013-07-12 09:08:11,397 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
> 2013-07-12 09:08:11,399 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 11
> 2013-07-12 09:08:11,400 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 13
> 2013-07-12 09:09:11,441 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 13
>
>
> And after sometime it said:
>
> 2013-07-12 11:03:19,799 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 795
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 12
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 797
> 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:26,003 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Entering safe mode.
> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ON.
> Resources are low on NN. Please add or free up more resources then turn
> off safe mode manually.  NOTE:  If you turn off safe mode before adding
> resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:31,004 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:31,005 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Already in safe mode.
> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ONResources are low on NN. Please add or free up more
> resources then turn off safe mode manually.  NOTE:  If you turn off safe
> mode before adding resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:36,005 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
>
> Thanks,
> Kishore
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Yes Harsh, I haven't set dfs.namenode.name.dir anywhere in config files. My
name node has again gone into safe mode today while it was idle. I shall
try setting this value to something other than /tmp



On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:

> > 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
> This is interesting. Its calling your volume "null", which may be more
> of a superficial bug.
>
> What is your dfs.namenode.name.dir set to? From
> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
> is being used off of the out-of-box defaults. Could you try to set it
> to a specific directory thats not on /tmp?
>
> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > I don't have it in my hdfs-site.xml, in which case probably the default
> > value is taken..
> >
> >
> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
> >>
> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
> >>
> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
> wrote:
> >>>
> >>> Hi Krishna,
> >>>
> >>>    Can you please send screenshots of namenode web UI.
> >>>
> >>> Thanks Aditya.
> >>>
> >>>
> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
> >>> <wr...@gmail.com> wrote:
> >>>>
> >>>> I have had enough space on the disk that is used, like around 30 Gigs
> >>>>
> >>>> Thanks,
> >>>> Kishore
> >>>>
> >>>>
> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
> >>>> <ve...@cloudwick.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> pls see the available space for NN storage directory.
> >>>>>
> >>>>> Thanks & Regards
> >>>>>
> >>>>> Venkat
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
> >>>>> <wr...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>  I am doing no activity on my single node cluster which is using
> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
> itself after
> >>>>>> a while. I was looking at the name node log and see many of these
> kinds of
> >>>>>> entries.. Can anything be interpreted from these?
> >>>>>>
> >>>>>> 2013-07-12 09:06:11,256 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 561
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 561
> >>>>>> 2013-07-12 09:07:11,291 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
> >>>>>> 2013-07-12 09:07:11,292 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
> >>>>>> 2013-07-12 09:07:11,293 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> >>>>>> 2013-07-12 09:07:11,294 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 563
> >>>>>> 2013-07-12 09:08:11,397 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 563
> >>>>>> 2013-07-12 09:08:11,399 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
> >>>>>> 2013-07-12 09:08:11,400 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
> >>>>>> 2013-07-12 09:09:11,441 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
> >>>>>>
> >>>>>>
> >>>>>> And after sometime it said:
> >>>>>>
> >>>>>> 2013-07-12 11:03:19,799 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 795
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 795
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 797
> >>>>>> 2013-07-12 11:04:26,002 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:26,003 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Entering safe mode.
> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ON.
> >>>>>> Resources are low on NN. Please add or free up more resources then
> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
> before adding
> >>>>>> resources, the NN will immediately return to safe mode..
> >>>>>> 2013-07-12 11:04:31,004 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:31,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Already in safe mode.
> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
> up more
> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off
> safe
> >>>>>> mode before adding resources, the NN will immediately return to
> safe mode..
> >>>>>> 2013-07-12 11:04:36,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Kishore
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> N Venkata Rami Reddy
> >>>>> Hadoop Admin
> >>>>> Cloudwick Technologies
> >>>>
> >>>>
> >>>
> >
>
>
>
> --
> Harsh J
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Harsh J <ha...@cloudera.com>.

Yeah I believe your /tmp was probably misbehaving somehow (running out
of space or otherwise). You could log a JIRA for the "null" seen in
the log though, it shouldn't have done that and should've shown the
real mount point.

On Fri, Jul 19, 2013 at 8:47 PM, Azuryy Yu <az...@gmail.com> wrote:
> this is not a bug.
>
> it has been documented.
>
> On Jul 19, 2013 10:13 PM, "Krishna Kishore Bonagiri"
> <wr...@gmail.com> wrote:
>>
>> Hi Harsh,
>>
>>   I have made my dfs.namenode.name.dir point to a subdirectory of my home,
>> and I don't see this issue again. So, is this a bug that we need to log into
>> JIRA?
>>
>> Thanks,
>> Kishore
>>
>>
>> On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>> > 2013-07-12 11:04:26,002 WARN
>>> > org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> > available on volume 'null' is 0, which is below the configured reserved
>>> > amount 104857600
>>>
>>> This is interesting. Its calling your volume "null", which may be more
>>> of a superficial bug.
>>>
>>> What is your dfs.namenode.name.dir set to? From
>>> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
>>> is being used off of the out-of-box defaults. Could you try to set it
>>> to a specific directory thats not on /tmp?
>>>
>>> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
>>> <wr...@gmail.com> wrote:
>>> > I don't have it in my hdfs-site.xml, in which case probably the default
>>> > value is taken..
>>> >
>>> >
>>> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
>>> >>
>>> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
>>> >>
>>> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Hi Krishna,
>>> >>>
>>> >>>    Can you please send screenshots of namenode web UI.
>>> >>>
>>> >>> Thanks Aditya.
>>> >>>
>>> >>>
>>> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
>>> >>> <wr...@gmail.com> wrote:
>>> >>>>
>>> >>>> I have had enough space on the disk that is used, like around 30
>>> >>>> Gigs
>>> >>>>
>>> >>>> Thanks,
>>> >>>> Kishore
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
>>> >>>> <ve...@cloudwick.com> wrote:
>>> >>>>>
>>> >>>>> Hi,
>>> >>>>> pls see the available space for NN storage directory.
>>> >>>>>
>>> >>>>> Thanks & Regards
>>> >>>>>
>>> >>>>> Venkat
>>> >>>>>
>>> >>>>>
>>> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
>>> >>>>> <wr...@gmail.com> wrote:
>>> >>>>>>
>>> >>>>>> Hi,
>>> >>>>>>
>>> >>>>>>  I am doing no activity on my single node cluster which is using
>>> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
>>> >>>>>> itself after
>>> >>>>>> a while. I was looking at the name node log and see many of these
>>> >>>>>> kinds of
>>> >>>>>> entries.. Can anything be interpreted from these?
>>> >>>>>>
>>> >>>>>> 2013-07-12 09:06:11,256 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 561
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 561
>>> >>>>>> 2013-07-12 09:07:11,291 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>> >>>>>> 2013-07-12 09:07:11,292 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>> >>>>>> 2013-07-12 09:07:11,293 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>> >>>>>> 2013-07-12 09:07:11,294 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 563
>>> >>>>>> 2013-07-12 09:08:11,397 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:08:11,398 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:08:11,398 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 563
>>> >>>>>> 2013-07-12 09:08:11,399 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>> >>>>>> 2013-07-12 09:08:11,400 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 09:08:11,402 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>> >>>>>> 2013-07-12 09:08:11,402 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 565
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 565
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>> >>>>>> 2013-07-12 09:09:11,441 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> And after sometime it said:
>>> >>>>>>
>>> >>>>>> 2013-07-12 11:03:19,799 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 795
>>> >>>>>> 2013-07-12 11:04:19,826 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 11:04:19,826 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 795
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 11:04:19,829 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>> >>>>>> 2013-07-12 11:04:19,829 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 797
>>> >>>>>> 2013-07-12 11:04:26,002 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>> 2013-07-12 11:04:26,003 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>>> >>>>>> on
>>> >>>>>> available disk space. Entering safe mode.
>>> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>>> >>>>>> STATE* Safe mode is ON.
>>> >>>>>> Resources are low on NN. Please add or free up more resources then
>>> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
>>> >>>>>> before adding
>>> >>>>>> resources, the NN will immediately return to safe mode..
>>> >>>>>> 2013-07-12 11:04:31,004 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>> 2013-07-12 11:04:31,005 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>>> >>>>>> on
>>> >>>>>> available disk space. Already in safe mode.
>>> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>>> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
>>> >>>>>> up more
>>> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn
>>> >>>>>> off safe
>>> >>>>>> mode before adding resources, the NN will immediately return to
>>> >>>>>> safe mode..
>>> >>>>>> 2013-07-12 11:04:36,005 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> Thanks,
>>> >>>>>> Kishore
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> N Venkata Rami Reddy
>>> >>>>> Hadoop Admin
>>> >>>>> Cloudwick Technologies
>>> >>>>
>>> >>>>
>>> >>>
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>



-- 
Harsh J

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Harsh J <ha...@cloudera.com>.

Yeah I believe your /tmp was probably misbehaving somehow (running out
of space or otherwise). You could log a JIRA for the "null" seen in
the log though, it shouldn't have done that and should've shown the
real mount point.

On Fri, Jul 19, 2013 at 8:47 PM, Azuryy Yu <az...@gmail.com> wrote:
> this is not a bug.
>
> it has been documented.
>
> On Jul 19, 2013 10:13 PM, "Krishna Kishore Bonagiri"
> <wr...@gmail.com> wrote:
>>
>> Hi Harsh,
>>
>>   I have made my dfs.namenode.name.dir point to a subdirectory of my home,
>> and I don't see this issue again. So, is this a bug that we need to log into
>> JIRA?
>>
>> Thanks,
>> Kishore
>>
>>
>> On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>> > 2013-07-12 11:04:26,002 WARN
>>> > org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> > available on volume 'null' is 0, which is below the configured reserved
>>> > amount 104857600
>>>
>>> This is interesting. Its calling your volume "null", which may be more
>>> of a superficial bug.
>>>
>>> What is your dfs.namenode.name.dir set to? From
>>> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
>>> is being used off of the out-of-box defaults. Could you try to set it
>>> to a specific directory thats not on /tmp?
>>>
>>> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
>>> <wr...@gmail.com> wrote:
>>> > I don't have it in my hdfs-site.xml, in which case probably the default
>>> > value is taken..
>>> >
>>> >
>>> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
>>> >>
>>> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
>>> >>
>>> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Hi Krishna,
>>> >>>
>>> >>>    Can you please send screenshots of namenode web UI.
>>> >>>
>>> >>> Thanks Aditya.
>>> >>>
>>> >>>
>>> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
>>> >>> <wr...@gmail.com> wrote:
>>> >>>>
>>> >>>> I have had enough space on the disk that is used, like around 30
>>> >>>> Gigs
>>> >>>>
>>> >>>> Thanks,
>>> >>>> Kishore
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
>>> >>>> <ve...@cloudwick.com> wrote:
>>> >>>>>
>>> >>>>> Hi,
>>> >>>>> pls see the available space for NN storage directory.
>>> >>>>>
>>> >>>>> Thanks & Regards
>>> >>>>>
>>> >>>>> Venkat
>>> >>>>>
>>> >>>>>
>>> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
>>> >>>>> <wr...@gmail.com> wrote:
>>> >>>>>>
>>> >>>>>> Hi,
>>> >>>>>>
>>> >>>>>>  I am doing no activity on my single node cluster which is using
>>> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
>>> >>>>>> itself after
>>> >>>>>> a while. I was looking at the name node log and see many of these
>>> >>>>>> kinds of
>>> >>>>>> entries.. Can anything be interpreted from these?
>>> >>>>>>
>>> >>>>>> 2013-07-12 09:06:11,256 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 561
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 561
>>> >>>>>> 2013-07-12 09:07:11,291 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>> >>>>>> 2013-07-12 09:07:11,292 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>> >>>>>> 2013-07-12 09:07:11,293 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>> >>>>>> 2013-07-12 09:07:11,294 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 563
>>> >>>>>> 2013-07-12 09:08:11,397 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:08:11,398 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:08:11,398 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 563
>>> >>>>>> 2013-07-12 09:08:11,399 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>> >>>>>> 2013-07-12 09:08:11,400 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 09:08:11,402 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>> >>>>>> 2013-07-12 09:08:11,402 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 565
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 565
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>> >>>>>> 2013-07-12 09:09:11,441 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> And after sometime it said:
>>> >>>>>>
>>> >>>>>> 2013-07-12 11:03:19,799 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 795
>>> >>>>>> 2013-07-12 11:04:19,826 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 11:04:19,826 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 795
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 11:04:19,829 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>> >>>>>> 2013-07-12 11:04:19,829 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 797
>>> >>>>>> 2013-07-12 11:04:26,002 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>> 2013-07-12 11:04:26,003 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>>> >>>>>> on
>>> >>>>>> available disk space. Entering safe mode.
>>> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>>> >>>>>> STATE* Safe mode is ON.
>>> >>>>>> Resources are low on NN. Please add or free up more resources then
>>> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
>>> >>>>>> before adding
>>> >>>>>> resources, the NN will immediately return to safe mode..
>>> >>>>>> 2013-07-12 11:04:31,004 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>> 2013-07-12 11:04:31,005 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>>> >>>>>> on
>>> >>>>>> available disk space. Already in safe mode.
>>> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>>> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
>>> >>>>>> up more
>>> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn
>>> >>>>>> off safe
>>> >>>>>> mode before adding resources, the NN will immediately return to
>>> >>>>>> safe mode..
>>> >>>>>> 2013-07-12 11:04:36,005 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> Thanks,
>>> >>>>>> Kishore
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> N Venkata Rami Reddy
>>> >>>>> Hadoop Admin
>>> >>>>> Cloudwick Technologies
>>> >>>>
>>> >>>>
>>> >>>
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>



-- 
Harsh J

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Harsh J <ha...@cloudera.com>.

Yeah I believe your /tmp was probably misbehaving somehow (running out
of space or otherwise). You could log a JIRA for the "null" seen in
the log though, it shouldn't have done that and should've shown the
real mount point.

On Fri, Jul 19, 2013 at 8:47 PM, Azuryy Yu <az...@gmail.com> wrote:
> this is not a bug.
>
> it has been documented.
>
> On Jul 19, 2013 10:13 PM, "Krishna Kishore Bonagiri"
> <wr...@gmail.com> wrote:
>>
>> Hi Harsh,
>>
>>   I have made my dfs.namenode.name.dir point to a subdirectory of my home,
>> and I don't see this issue again. So, is this a bug that we need to log into
>> JIRA?
>>
>> Thanks,
>> Kishore
>>
>>
>> On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>> > 2013-07-12 11:04:26,002 WARN
>>> > org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> > available on volume 'null' is 0, which is below the configured reserved
>>> > amount 104857600
>>>
>>> This is interesting. Its calling your volume "null", which may be more
>>> of a superficial bug.
>>>
>>> What is your dfs.namenode.name.dir set to? From
>>> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
>>> is being used off of the out-of-box defaults. Could you try to set it
>>> to a specific directory thats not on /tmp?
>>>
>>> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
>>> <wr...@gmail.com> wrote:
>>> > I don't have it in my hdfs-site.xml, in which case probably the default
>>> > value is taken..
>>> >
>>> >
>>> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
>>> >>
>>> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
>>> >>
>>> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Hi Krishna,
>>> >>>
>>> >>>    Can you please send screenshots of namenode web UI.
>>> >>>
>>> >>> Thanks Aditya.
>>> >>>
>>> >>>
>>> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
>>> >>> <wr...@gmail.com> wrote:
>>> >>>>
>>> >>>> I have had enough space on the disk that is used, like around 30
>>> >>>> Gigs
>>> >>>>
>>> >>>> Thanks,
>>> >>>> Kishore
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
>>> >>>> <ve...@cloudwick.com> wrote:
>>> >>>>>
>>> >>>>> Hi,
>>> >>>>> pls see the available space for NN storage directory.
>>> >>>>>
>>> >>>>> Thanks & Regards
>>> >>>>>
>>> >>>>> Venkat
>>> >>>>>
>>> >>>>>
>>> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
>>> >>>>> <wr...@gmail.com> wrote:
>>> >>>>>>
>>> >>>>>> Hi,
>>> >>>>>>
>>> >>>>>>  I am doing no activity on my single node cluster which is using
>>> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
>>> >>>>>> itself after
>>> >>>>>> a while. I was looking at the name node log and see many of these
>>> >>>>>> kinds of
>>> >>>>>> entries.. Can anything be interpreted from these?
>>> >>>>>>
>>> >>>>>> 2013-07-12 09:06:11,256 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 561
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 561
>>> >>>>>> 2013-07-12 09:07:11,291 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>> >>>>>> 2013-07-12 09:07:11,292 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>> >>>>>> 2013-07-12 09:07:11,293 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>> >>>>>> 2013-07-12 09:07:11,294 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 563
>>> >>>>>> 2013-07-12 09:08:11,397 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:08:11,398 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:08:11,398 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 563
>>> >>>>>> 2013-07-12 09:08:11,399 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>> >>>>>> 2013-07-12 09:08:11,400 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 09:08:11,402 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>> >>>>>> 2013-07-12 09:08:11,402 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 565
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 565
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>> >>>>>> 2013-07-12 09:09:11,441 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> And after sometime it said:
>>> >>>>>>
>>> >>>>>> 2013-07-12 11:03:19,799 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 795
>>> >>>>>> 2013-07-12 11:04:19,826 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 11:04:19,826 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 795
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 11:04:19,829 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>> >>>>>> 2013-07-12 11:04:19,829 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 797
>>> >>>>>> 2013-07-12 11:04:26,002 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>> 2013-07-12 11:04:26,003 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>>> >>>>>> on
>>> >>>>>> available disk space. Entering safe mode.
>>> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>>> >>>>>> STATE* Safe mode is ON.
>>> >>>>>> Resources are low on NN. Please add or free up more resources then
>>> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
>>> >>>>>> before adding
>>> >>>>>> resources, the NN will immediately return to safe mode..
>>> >>>>>> 2013-07-12 11:04:31,004 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>> 2013-07-12 11:04:31,005 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>>> >>>>>> on
>>> >>>>>> available disk space. Already in safe mode.
>>> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>>> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
>>> >>>>>> up more
>>> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn
>>> >>>>>> off safe
>>> >>>>>> mode before adding resources, the NN will immediately return to
>>> >>>>>> safe mode..
>>> >>>>>> 2013-07-12 11:04:36,005 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> Thanks,
>>> >>>>>> Kishore
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> N Venkata Rami Reddy
>>> >>>>> Hadoop Admin
>>> >>>>> Cloudwick Technologies
>>> >>>>
>>> >>>>
>>> >>>
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>



-- 
Harsh J

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Harsh J <ha...@cloudera.com>.

Yeah I believe your /tmp was probably misbehaving somehow (running out
of space or otherwise). You could log a JIRA for the "null" seen in
the log though, it shouldn't have done that and should've shown the
real mount point.

On Fri, Jul 19, 2013 at 8:47 PM, Azuryy Yu <az...@gmail.com> wrote:
> this is not a bug.
>
> it has been documented.
>
> On Jul 19, 2013 10:13 PM, "Krishna Kishore Bonagiri"
> <wr...@gmail.com> wrote:
>>
>> Hi Harsh,
>>
>>   I have made my dfs.namenode.name.dir point to a subdirectory of my home,
>> and I don't see this issue again. So, is this a bug that we need to log into
>> JIRA?
>>
>> Thanks,
>> Kishore
>>
>>
>> On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>> > 2013-07-12 11:04:26,002 WARN
>>> > org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> > available on volume 'null' is 0, which is below the configured reserved
>>> > amount 104857600
>>>
>>> This is interesting. Its calling your volume "null", which may be more
>>> of a superficial bug.
>>>
>>> What is your dfs.namenode.name.dir set to? From
>>> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
>>> is being used off of the out-of-box defaults. Could you try to set it
>>> to a specific directory thats not on /tmp?
>>>
>>> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
>>> <wr...@gmail.com> wrote:
>>> > I don't have it in my hdfs-site.xml, in which case probably the default
>>> > value is taken..
>>> >
>>> >
>>> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
>>> >>
>>> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
>>> >>
>>> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Hi Krishna,
>>> >>>
>>> >>>    Can you please send screenshots of namenode web UI.
>>> >>>
>>> >>> Thanks Aditya.
>>> >>>
>>> >>>
>>> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
>>> >>> <wr...@gmail.com> wrote:
>>> >>>>
>>> >>>> I have had enough space on the disk that is used, like around 30
>>> >>>> Gigs
>>> >>>>
>>> >>>> Thanks,
>>> >>>> Kishore
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
>>> >>>> <ve...@cloudwick.com> wrote:
>>> >>>>>
>>> >>>>> Hi,
>>> >>>>> pls see the available space for NN storage directory.
>>> >>>>>
>>> >>>>> Thanks & Regards
>>> >>>>>
>>> >>>>> Venkat
>>> >>>>>
>>> >>>>>
>>> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
>>> >>>>> <wr...@gmail.com> wrote:
>>> >>>>>>
>>> >>>>>> Hi,
>>> >>>>>>
>>> >>>>>>  I am doing no activity on my single node cluster which is using
>>> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
>>> >>>>>> itself after
>>> >>>>>> a while. I was looking at the name node log and see many of these
>>> >>>>>> kinds of
>>> >>>>>> entries.. Can anything be interpreted from these?
>>> >>>>>>
>>> >>>>>> 2013-07-12 09:06:11,256 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 561
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:07:11,290 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 561
>>> >>>>>> 2013-07-12 09:07:11,291 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>> >>>>>> 2013-07-12 09:07:11,292 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>> >>>>>> 2013-07-12 09:07:11,293 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>> >>>>>> 2013-07-12 09:07:11,294 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 563
>>> >>>>>> 2013-07-12 09:08:11,397 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:08:11,398 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:08:11,398 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 563
>>> >>>>>> 2013-07-12 09:08:11,399 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>> >>>>>> 2013-07-12 09:08:11,400 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 09:08:11,402 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>> >>>>>> 2013-07-12 09:08:11,402 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 565
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 565
>>> >>>>>> 2013-07-12 09:09:11,440 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>> >>>>>> 2013-07-12 09:09:11,441 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> And after sometime it said:
>>> >>>>>>
>>> >>>>>> 2013-07-12 11:03:19,799 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 795
>>> >>>>>> 2013-07-12 11:04:19,826 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> >>>>>> from
>>> >>>>>> 9.70.137.114
>>> >>>>>> 2013-07-12 11:04:19,826 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit
>>> >>>>>> logs
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>>> >>>>>> segment 795
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 11:04:19,827 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>>> >>>>>> transactions: 2
>>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>>> >>>>>> in Syncs:
>>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> >>>>>> 2013-07-12 11:04:19,829 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>>> >>>>>> Finalizing edits
>>> >>>>>> file
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
>>> >>>>>> ->
>>> >>>>>>
>>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>> >>>>>> 2013-07-12 11:04:19,829 INFO
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>>> >>>>>> segment at
>>> >>>>>> 797
>>> >>>>>> 2013-07-12 11:04:26,002 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>> 2013-07-12 11:04:26,003 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>>> >>>>>> on
>>> >>>>>> available disk space. Entering safe mode.
>>> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>>> >>>>>> STATE* Safe mode is ON.
>>> >>>>>> Resources are low on NN. Please add or free up more resources then
>>> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
>>> >>>>>> before adding
>>> >>>>>> resources, the NN will immediately return to safe mode..
>>> >>>>>> 2013-07-12 11:04:31,004 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>> 2013-07-12 11:04:31,005 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>>> >>>>>> on
>>> >>>>>> available disk space. Already in safe mode.
>>> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>>> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
>>> >>>>>> up more
>>> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn
>>> >>>>>> off safe
>>> >>>>>> mode before adding resources, the NN will immediately return to
>>> >>>>>> safe mode..
>>> >>>>>> 2013-07-12 11:04:36,005 WARN
>>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>>> >>>>>> Space
>>> >>>>>> available on volume 'null' is 0, which is below the configured
>>> >>>>>> reserved
>>> >>>>>> amount 104857600
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> Thanks,
>>> >>>>>> Kishore
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> N Venkata Rami Reddy
>>> >>>>> Hadoop Admin
>>> >>>>> Cloudwick Technologies
>>> >>>>
>>> >>>>
>>> >>>
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>



-- 
Harsh J

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Azuryy Yu <az...@gmail.com>.

this is not a bug.

it has been documented.
 On Jul 19, 2013 10:13 PM, "Krishna Kishore Bonagiri" <
write2kishore@gmail.com> wrote:

> Hi Harsh,
>
>   I have made my dfs.namenode.name.dir point to a subdirectory of my
> home, and I don't see this issue again. So, is this a bug that we need to
> log into JIRA?
>
> Thanks,
> Kishore
>
>
> On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> > 2013-07-12 11:04:26,002 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>>
>> This is interesting. Its calling your volume "null", which may be more
>> of a superficial bug.
>>
>> What is your dfs.namenode.name.dir set to? From
>> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
>> is being used off of the out-of-box defaults. Could you try to set it
>> to a specific directory thats not on /tmp?
>>
>> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
>> <wr...@gmail.com> wrote:
>> > I don't have it in my hdfs-site.xml, in which case probably the default
>> > value is taken..
>> >
>> >
>> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
>> >>
>> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
>> >>
>> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
>> wrote:
>> >>>
>> >>> Hi Krishna,
>> >>>
>> >>>    Can you please send screenshots of namenode web UI.
>> >>>
>> >>> Thanks Aditya.
>> >>>
>> >>>
>> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
>> >>> <wr...@gmail.com> wrote:
>> >>>>
>> >>>> I have had enough space on the disk that is used, like around 30 Gigs
>> >>>>
>> >>>> Thanks,
>> >>>> Kishore
>> >>>>
>> >>>>
>> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
>> >>>> <ve...@cloudwick.com> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>> pls see the available space for NN storage directory.
>> >>>>>
>> >>>>> Thanks & Regards
>> >>>>>
>> >>>>> Venkat
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
>> >>>>> <wr...@gmail.com> wrote:
>> >>>>>>
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>>  I am doing no activity on my single node cluster which is using
>> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
>> itself after
>> >>>>>> a while. I was looking at the name node log and see many of these
>> kinds of
>> >>>>>> entries.. Can anything be interpreted from these?
>> >>>>>>
>> >>>>>> 2013-07-12 09:06:11,256 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 561
>> >>>>>> 2013-07-12 09:07:11,290 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 09:07:11,290 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 09:07:11,290 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 561
>> >>>>>> 2013-07-12 09:07:11,291 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>> >>>>>> 2013-07-12 09:07:11,292 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>> >>>>>> 2013-07-12 09:07:11,293 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>> Finalizing edits
>> >>>>>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
>> >>>>>> ->
>> >>>>>>
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>> >>>>>> 2013-07-12 09:07:11,294 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 563
>> >>>>>> 2013-07-12 09:08:11,397 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 09:08:11,398 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 09:08:11,398 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 563
>> >>>>>> 2013-07-12 09:08:11,399 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>> >>>>>> 2013-07-12 09:08:11,400 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> >>>>>> 2013-07-12 09:08:11,402 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>> Finalizing edits
>> >>>>>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
>> >>>>>> ->
>> >>>>>>
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>> >>>>>> 2013-07-12 09:08:11,402 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 565
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 565
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>> >>>>>> 2013-07-12 09:09:11,441 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>> >>>>>>
>> >>>>>>
>> >>>>>> And after sometime it said:
>> >>>>>>
>> >>>>>> 2013-07-12 11:03:19,799 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 795
>> >>>>>> 2013-07-12 11:04:19,826 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 11:04:19,826 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 11:04:19,827 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 795
>> >>>>>> 2013-07-12 11:04:19,827 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>> >>>>>> 2013-07-12 11:04:19,827 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> >>>>>> 2013-07-12 11:04:19,829 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>> Finalizing edits
>> >>>>>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
>> >>>>>> ->
>> >>>>>>
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>> >>>>>> 2013-07-12 11:04:19,829 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 797
>> >>>>>> 2013-07-12 11:04:26,002 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>> Space
>> >>>>>> available on volume 'null' is 0, which is below the configured
>> reserved
>> >>>>>> amount 104857600
>> >>>>>> 2013-07-12 11:04:26,003 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>> on
>> >>>>>> available disk space. Entering safe mode.
>> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>> >>>>>> STATE* Safe mode is ON.
>> >>>>>> Resources are low on NN. Please add or free up more resources then
>> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
>> before adding
>> >>>>>> resources, the NN will immediately return to safe mode..
>> >>>>>> 2013-07-12 11:04:31,004 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>> Space
>> >>>>>> available on volume 'null' is 0, which is below the configured
>> reserved
>> >>>>>> amount 104857600
>> >>>>>> 2013-07-12 11:04:31,005 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>> on
>> >>>>>> available disk space. Already in safe mode.
>> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
>> up more
>> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn
>> off safe
>> >>>>>> mode before adding resources, the NN will immediately return to
>> safe mode..
>> >>>>>> 2013-07-12 11:04:36,005 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>> Space
>> >>>>>> available on volume 'null' is 0, which is below the configured
>> reserved
>> >>>>>> amount 104857600
>> >>>>>>
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Kishore
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> N Venkata Rami Reddy
>> >>>>> Hadoop Admin
>> >>>>> Cloudwick Technologies
>> >>>>
>> >>>>
>> >>>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Azuryy Yu <az...@gmail.com>.

this is not a bug.

it has been documented.
 On Jul 19, 2013 10:13 PM, "Krishna Kishore Bonagiri" <
write2kishore@gmail.com> wrote:

> Hi Harsh,
>
>   I have made my dfs.namenode.name.dir point to a subdirectory of my
> home, and I don't see this issue again. So, is this a bug that we need to
> log into JIRA?
>
> Thanks,
> Kishore
>
>
> On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> > 2013-07-12 11:04:26,002 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>>
>> This is interesting. Its calling your volume "null", which may be more
>> of a superficial bug.
>>
>> What is your dfs.namenode.name.dir set to? From
>> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
>> is being used off of the out-of-box defaults. Could you try to set it
>> to a specific directory thats not on /tmp?
>>
>> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
>> <wr...@gmail.com> wrote:
>> > I don't have it in my hdfs-site.xml, in which case probably the default
>> > value is taken..
>> >
>> >
>> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
>> >>
>> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
>> >>
>> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
>> wrote:
>> >>>
>> >>> Hi Krishna,
>> >>>
>> >>>    Can you please send screenshots of namenode web UI.
>> >>>
>> >>> Thanks Aditya.
>> >>>
>> >>>
>> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
>> >>> <wr...@gmail.com> wrote:
>> >>>>
>> >>>> I have had enough space on the disk that is used, like around 30 Gigs
>> >>>>
>> >>>> Thanks,
>> >>>> Kishore
>> >>>>
>> >>>>
>> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
>> >>>> <ve...@cloudwick.com> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>> pls see the available space for NN storage directory.
>> >>>>>
>> >>>>> Thanks & Regards
>> >>>>>
>> >>>>> Venkat
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
>> >>>>> <wr...@gmail.com> wrote:
>> >>>>>>
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>>  I am doing no activity on my single node cluster which is using
>> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
>> itself after
>> >>>>>> a while. I was looking at the name node log and see many of these
>> kinds of
>> >>>>>> entries.. Can anything be interpreted from these?
>> >>>>>>
>> >>>>>> 2013-07-12 09:06:11,256 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 561
>> >>>>>> 2013-07-12 09:07:11,290 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 09:07:11,290 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 09:07:11,290 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 561
>> >>>>>> 2013-07-12 09:07:11,291 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>> >>>>>> 2013-07-12 09:07:11,292 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>> >>>>>> 2013-07-12 09:07:11,293 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>> Finalizing edits
>> >>>>>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
>> >>>>>> ->
>> >>>>>>
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>> >>>>>> 2013-07-12 09:07:11,294 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 563
>> >>>>>> 2013-07-12 09:08:11,397 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 09:08:11,398 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 09:08:11,398 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 563
>> >>>>>> 2013-07-12 09:08:11,399 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>> >>>>>> 2013-07-12 09:08:11,400 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> >>>>>> 2013-07-12 09:08:11,402 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>> Finalizing edits
>> >>>>>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
>> >>>>>> ->
>> >>>>>>
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>> >>>>>> 2013-07-12 09:08:11,402 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 565
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 565
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>> >>>>>> 2013-07-12 09:09:11,441 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>> >>>>>>
>> >>>>>>
>> >>>>>> And after sometime it said:
>> >>>>>>
>> >>>>>> 2013-07-12 11:03:19,799 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 795
>> >>>>>> 2013-07-12 11:04:19,826 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 11:04:19,826 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 11:04:19,827 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 795
>> >>>>>> 2013-07-12 11:04:19,827 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>> >>>>>> 2013-07-12 11:04:19,827 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> >>>>>> 2013-07-12 11:04:19,829 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>> Finalizing edits
>> >>>>>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
>> >>>>>> ->
>> >>>>>>
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>> >>>>>> 2013-07-12 11:04:19,829 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 797
>> >>>>>> 2013-07-12 11:04:26,002 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>> Space
>> >>>>>> available on volume 'null' is 0, which is below the configured
>> reserved
>> >>>>>> amount 104857600
>> >>>>>> 2013-07-12 11:04:26,003 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>> on
>> >>>>>> available disk space. Entering safe mode.
>> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>> >>>>>> STATE* Safe mode is ON.
>> >>>>>> Resources are low on NN. Please add or free up more resources then
>> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
>> before adding
>> >>>>>> resources, the NN will immediately return to safe mode..
>> >>>>>> 2013-07-12 11:04:31,004 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>> Space
>> >>>>>> available on volume 'null' is 0, which is below the configured
>> reserved
>> >>>>>> amount 104857600
>> >>>>>> 2013-07-12 11:04:31,005 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>> on
>> >>>>>> available disk space. Already in safe mode.
>> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
>> up more
>> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn
>> off safe
>> >>>>>> mode before adding resources, the NN will immediately return to
>> safe mode..
>> >>>>>> 2013-07-12 11:04:36,005 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>> Space
>> >>>>>> available on volume 'null' is 0, which is below the configured
>> reserved
>> >>>>>> amount 104857600
>> >>>>>>
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Kishore
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> N Venkata Rami Reddy
>> >>>>> Hadoop Admin
>> >>>>> Cloudwick Technologies
>> >>>>
>> >>>>
>> >>>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Azuryy Yu <az...@gmail.com>.

this is not a bug.

it has been documented.
 On Jul 19, 2013 10:13 PM, "Krishna Kishore Bonagiri" <
write2kishore@gmail.com> wrote:

> Hi Harsh,
>
>   I have made my dfs.namenode.name.dir point to a subdirectory of my
> home, and I don't see this issue again. So, is this a bug that we need to
> log into JIRA?
>
> Thanks,
> Kishore
>
>
> On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> > 2013-07-12 11:04:26,002 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>>
>> This is interesting. Its calling your volume "null", which may be more
>> of a superficial bug.
>>
>> What is your dfs.namenode.name.dir set to? From
>> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
>> is being used off of the out-of-box defaults. Could you try to set it
>> to a specific directory thats not on /tmp?
>>
>> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
>> <wr...@gmail.com> wrote:
>> > I don't have it in my hdfs-site.xml, in which case probably the default
>> > value is taken..
>> >
>> >
>> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
>> >>
>> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
>> >>
>> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
>> wrote:
>> >>>
>> >>> Hi Krishna,
>> >>>
>> >>>    Can you please send screenshots of namenode web UI.
>> >>>
>> >>> Thanks Aditya.
>> >>>
>> >>>
>> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
>> >>> <wr...@gmail.com> wrote:
>> >>>>
>> >>>> I have had enough space on the disk that is used, like around 30 Gigs
>> >>>>
>> >>>> Thanks,
>> >>>> Kishore
>> >>>>
>> >>>>
>> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
>> >>>> <ve...@cloudwick.com> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>> pls see the available space for NN storage directory.
>> >>>>>
>> >>>>> Thanks & Regards
>> >>>>>
>> >>>>> Venkat
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
>> >>>>> <wr...@gmail.com> wrote:
>> >>>>>>
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>>  I am doing no activity on my single node cluster which is using
>> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
>> itself after
>> >>>>>> a while. I was looking at the name node log and see many of these
>> kinds of
>> >>>>>> entries.. Can anything be interpreted from these?
>> >>>>>>
>> >>>>>> 2013-07-12 09:06:11,256 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 561
>> >>>>>> 2013-07-12 09:07:11,290 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 09:07:11,290 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 09:07:11,290 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 561
>> >>>>>> 2013-07-12 09:07:11,291 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>> >>>>>> 2013-07-12 09:07:11,292 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>> >>>>>> 2013-07-12 09:07:11,293 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>> Finalizing edits
>> >>>>>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
>> >>>>>> ->
>> >>>>>>
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>> >>>>>> 2013-07-12 09:07:11,294 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 563
>> >>>>>> 2013-07-12 09:08:11,397 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 09:08:11,398 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 09:08:11,398 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 563
>> >>>>>> 2013-07-12 09:08:11,399 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>> >>>>>> 2013-07-12 09:08:11,400 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> >>>>>> 2013-07-12 09:08:11,402 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>> Finalizing edits
>> >>>>>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
>> >>>>>> ->
>> >>>>>>
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>> >>>>>> 2013-07-12 09:08:11,402 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 565
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 565
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>> >>>>>> 2013-07-12 09:09:11,441 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>> >>>>>>
>> >>>>>>
>> >>>>>> And after sometime it said:
>> >>>>>>
>> >>>>>> 2013-07-12 11:03:19,799 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 795
>> >>>>>> 2013-07-12 11:04:19,826 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 11:04:19,826 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 11:04:19,827 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 795
>> >>>>>> 2013-07-12 11:04:19,827 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>> >>>>>> 2013-07-12 11:04:19,827 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> >>>>>> 2013-07-12 11:04:19,829 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>> Finalizing edits
>> >>>>>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
>> >>>>>> ->
>> >>>>>>
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>> >>>>>> 2013-07-12 11:04:19,829 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 797
>> >>>>>> 2013-07-12 11:04:26,002 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>> Space
>> >>>>>> available on volume 'null' is 0, which is below the configured
>> reserved
>> >>>>>> amount 104857600
>> >>>>>> 2013-07-12 11:04:26,003 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>> on
>> >>>>>> available disk space. Entering safe mode.
>> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>> >>>>>> STATE* Safe mode is ON.
>> >>>>>> Resources are low on NN. Please add or free up more resources then
>> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
>> before adding
>> >>>>>> resources, the NN will immediately return to safe mode..
>> >>>>>> 2013-07-12 11:04:31,004 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>> Space
>> >>>>>> available on volume 'null' is 0, which is below the configured
>> reserved
>> >>>>>> amount 104857600
>> >>>>>> 2013-07-12 11:04:31,005 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>> on
>> >>>>>> available disk space. Already in safe mode.
>> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
>> up more
>> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn
>> off safe
>> >>>>>> mode before adding resources, the NN will immediately return to
>> safe mode..
>> >>>>>> 2013-07-12 11:04:36,005 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>> Space
>> >>>>>> available on volume 'null' is 0, which is below the configured
>> reserved
>> >>>>>> amount 104857600
>> >>>>>>
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Kishore
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> N Venkata Rami Reddy
>> >>>>> Hadoop Admin
>> >>>>> Cloudwick Technologies
>> >>>>
>> >>>>
>> >>>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Azuryy Yu <az...@gmail.com>.

this is not a bug.

it has been documented.
 On Jul 19, 2013 10:13 PM, "Krishna Kishore Bonagiri" <
write2kishore@gmail.com> wrote:

> Hi Harsh,
>
>   I have made my dfs.namenode.name.dir point to a subdirectory of my
> home, and I don't see this issue again. So, is this a bug that we need to
> log into JIRA?
>
> Thanks,
> Kishore
>
>
> On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> > 2013-07-12 11:04:26,002 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>>
>> This is interesting. Its calling your volume "null", which may be more
>> of a superficial bug.
>>
>> What is your dfs.namenode.name.dir set to? From
>> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
>> is being used off of the out-of-box defaults. Could you try to set it
>> to a specific directory thats not on /tmp?
>>
>> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
>> <wr...@gmail.com> wrote:
>> > I don't have it in my hdfs-site.xml, in which case probably the default
>> > value is taken..
>> >
>> >
>> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
>> >>
>> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
>> >>
>> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
>> wrote:
>> >>>
>> >>> Hi Krishna,
>> >>>
>> >>>    Can you please send screenshots of namenode web UI.
>> >>>
>> >>> Thanks Aditya.
>> >>>
>> >>>
>> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
>> >>> <wr...@gmail.com> wrote:
>> >>>>
>> >>>> I have had enough space on the disk that is used, like around 30 Gigs
>> >>>>
>> >>>> Thanks,
>> >>>> Kishore
>> >>>>
>> >>>>
>> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
>> >>>> <ve...@cloudwick.com> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>> pls see the available space for NN storage directory.
>> >>>>>
>> >>>>> Thanks & Regards
>> >>>>>
>> >>>>> Venkat
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
>> >>>>> <wr...@gmail.com> wrote:
>> >>>>>>
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>>  I am doing no activity on my single node cluster which is using
>> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
>> itself after
>> >>>>>> a while. I was looking at the name node log and see many of these
>> kinds of
>> >>>>>> entries.. Can anything be interpreted from these?
>> >>>>>>
>> >>>>>> 2013-07-12 09:06:11,256 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 561
>> >>>>>> 2013-07-12 09:07:11,290 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 09:07:11,290 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 09:07:11,290 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 561
>> >>>>>> 2013-07-12 09:07:11,291 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>> >>>>>> 2013-07-12 09:07:11,292 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>> >>>>>> 2013-07-12 09:07:11,293 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>> Finalizing edits
>> >>>>>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
>> >>>>>> ->
>> >>>>>>
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>> >>>>>> 2013-07-12 09:07:11,294 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 563
>> >>>>>> 2013-07-12 09:08:11,397 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 09:08:11,398 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 09:08:11,398 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 563
>> >>>>>> 2013-07-12 09:08:11,399 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>> >>>>>> 2013-07-12 09:08:11,400 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> >>>>>> 2013-07-12 09:08:11,402 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>> Finalizing edits
>> >>>>>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
>> >>>>>> ->
>> >>>>>>
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>> >>>>>> 2013-07-12 09:08:11,402 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 565
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 565
>> >>>>>> 2013-07-12 09:09:11,440 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>> >>>>>> 2013-07-12 09:09:11,441 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>> >>>>>>
>> >>>>>>
>> >>>>>> And after sometime it said:
>> >>>>>>
>> >>>>>> 2013-07-12 11:03:19,799 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 795
>> >>>>>> 2013-07-12 11:04:19,826 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from
>> >>>>>> 9.70.137.114
>> >>>>>> 2013-07-12 11:04:19,826 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> >>>>>> 2013-07-12 11:04:19,827 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
>> segment 795
>> >>>>>> 2013-07-12 11:04:19,827 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>> >>>>>> 2013-07-12 11:04:19,827 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
>> transactions: 2
>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
>> in Syncs:
>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> >>>>>> 2013-07-12 11:04:19,829 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
>> Finalizing edits
>> >>>>>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
>> >>>>>> ->
>> >>>>>>
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>> >>>>>> 2013-07-12 11:04:19,829 INFO
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
>> segment at
>> >>>>>> 797
>> >>>>>> 2013-07-12 11:04:26,002 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>> Space
>> >>>>>> available on volume 'null' is 0, which is below the configured
>> reserved
>> >>>>>> amount 104857600
>> >>>>>> 2013-07-12 11:04:26,003 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>> on
>> >>>>>> available disk space. Entering safe mode.
>> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>> >>>>>> STATE* Safe mode is ON.
>> >>>>>> Resources are low on NN. Please add or free up more resources then
>> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
>> before adding
>> >>>>>> resources, the NN will immediately return to safe mode..
>> >>>>>> 2013-07-12 11:04:31,004 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>> Space
>> >>>>>> available on volume 'null' is 0, which is below the configured
>> reserved
>> >>>>>> amount 104857600
>> >>>>>> 2013-07-12 11:04:31,005 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low
>> on
>> >>>>>> available disk space. Already in safe mode.
>> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
>> up more
>> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn
>> off safe
>> >>>>>> mode before adding resources, the NN will immediately return to
>> safe mode..
>> >>>>>> 2013-07-12 11:04:36,005 WARN
>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
>> Space
>> >>>>>> available on volume 'null' is 0, which is below the configured
>> reserved
>> >>>>>> amount 104857600
>> >>>>>>
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Kishore
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> N Venkata Rami Reddy
>> >>>>> Hadoop Admin
>> >>>>> Cloudwick Technologies
>> >>>>
>> >>>>
>> >>>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Harsh,

  I have made my dfs.namenode.name.dir point to a subdirectory of my home,
and I don't see this issue again. So, is this a bug that we need to log
into JIRA?

Thanks,
Kishore


On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:

> > 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
> This is interesting. Its calling your volume "null", which may be more
> of a superficial bug.
>
> What is your dfs.namenode.name.dir set to? From
> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
> is being used off of the out-of-box defaults. Could you try to set it
> to a specific directory thats not on /tmp?
>
> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > I don't have it in my hdfs-site.xml, in which case probably the default
> > value is taken..
> >
> >
> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
> >>
> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
> >>
> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
> wrote:
> >>>
> >>> Hi Krishna,
> >>>
> >>>    Can you please send screenshots of namenode web UI.
> >>>
> >>> Thanks Aditya.
> >>>
> >>>
> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
> >>> <wr...@gmail.com> wrote:
> >>>>
> >>>> I have had enough space on the disk that is used, like around 30 Gigs
> >>>>
> >>>> Thanks,
> >>>> Kishore
> >>>>
> >>>>
> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
> >>>> <ve...@cloudwick.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> pls see the available space for NN storage directory.
> >>>>>
> >>>>> Thanks & Regards
> >>>>>
> >>>>> Venkat
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
> >>>>> <wr...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>  I am doing no activity on my single node cluster which is using
> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
> itself after
> >>>>>> a while. I was looking at the name node log and see many of these
> kinds of
> >>>>>> entries.. Can anything be interpreted from these?
> >>>>>>
> >>>>>> 2013-07-12 09:06:11,256 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 561
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 561
> >>>>>> 2013-07-12 09:07:11,291 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
> >>>>>> 2013-07-12 09:07:11,292 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
> >>>>>> 2013-07-12 09:07:11,293 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> >>>>>> 2013-07-12 09:07:11,294 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 563
> >>>>>> 2013-07-12 09:08:11,397 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 563
> >>>>>> 2013-07-12 09:08:11,399 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
> >>>>>> 2013-07-12 09:08:11,400 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
> >>>>>> 2013-07-12 09:09:11,441 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
> >>>>>>
> >>>>>>
> >>>>>> And after sometime it said:
> >>>>>>
> >>>>>> 2013-07-12 11:03:19,799 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 795
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 795
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 797
> >>>>>> 2013-07-12 11:04:26,002 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:26,003 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Entering safe mode.
> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ON.
> >>>>>> Resources are low on NN. Please add or free up more resources then
> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
> before adding
> >>>>>> resources, the NN will immediately return to safe mode..
> >>>>>> 2013-07-12 11:04:31,004 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:31,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Already in safe mode.
> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
> up more
> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off
> safe
> >>>>>> mode before adding resources, the NN will immediately return to
> safe mode..
> >>>>>> 2013-07-12 11:04:36,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Kishore
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> N Venkata Rami Reddy
> >>>>> Hadoop Admin
> >>>>> Cloudwick Technologies
> >>>>
> >>>>
> >>>
> >
>
>
>
> --
> Harsh J
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Harsh,

  I have made my dfs.namenode.name.dir point to a subdirectory of my home,
and I don't see this issue again. So, is this a bug that we need to log
into JIRA?

Thanks,
Kishore


On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:

> > 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
> This is interesting. Its calling your volume "null", which may be more
> of a superficial bug.
>
> What is your dfs.namenode.name.dir set to? From
> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
> is being used off of the out-of-box defaults. Could you try to set it
> to a specific directory thats not on /tmp?
>
> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > I don't have it in my hdfs-site.xml, in which case probably the default
> > value is taken..
> >
> >
> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
> >>
> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
> >>
> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
> wrote:
> >>>
> >>> Hi Krishna,
> >>>
> >>>    Can you please send screenshots of namenode web UI.
> >>>
> >>> Thanks Aditya.
> >>>
> >>>
> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
> >>> <wr...@gmail.com> wrote:
> >>>>
> >>>> I have had enough space on the disk that is used, like around 30 Gigs
> >>>>
> >>>> Thanks,
> >>>> Kishore
> >>>>
> >>>>
> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
> >>>> <ve...@cloudwick.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> pls see the available space for NN storage directory.
> >>>>>
> >>>>> Thanks & Regards
> >>>>>
> >>>>> Venkat
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
> >>>>> <wr...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>  I am doing no activity on my single node cluster which is using
> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
> itself after
> >>>>>> a while. I was looking at the name node log and see many of these
> kinds of
> >>>>>> entries.. Can anything be interpreted from these?
> >>>>>>
> >>>>>> 2013-07-12 09:06:11,256 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 561
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 561
> >>>>>> 2013-07-12 09:07:11,291 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
> >>>>>> 2013-07-12 09:07:11,292 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
> >>>>>> 2013-07-12 09:07:11,293 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> >>>>>> 2013-07-12 09:07:11,294 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 563
> >>>>>> 2013-07-12 09:08:11,397 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 563
> >>>>>> 2013-07-12 09:08:11,399 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
> >>>>>> 2013-07-12 09:08:11,400 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
> >>>>>> 2013-07-12 09:09:11,441 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
> >>>>>>
> >>>>>>
> >>>>>> And after sometime it said:
> >>>>>>
> >>>>>> 2013-07-12 11:03:19,799 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 795
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 795
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 797
> >>>>>> 2013-07-12 11:04:26,002 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:26,003 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Entering safe mode.
> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ON.
> >>>>>> Resources are low on NN. Please add or free up more resources then
> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
> before adding
> >>>>>> resources, the NN will immediately return to safe mode..
> >>>>>> 2013-07-12 11:04:31,004 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:31,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Already in safe mode.
> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
> up more
> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off
> safe
> >>>>>> mode before adding resources, the NN will immediately return to
> safe mode..
> >>>>>> 2013-07-12 11:04:36,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Kishore
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> N Venkata Rami Reddy
> >>>>> Hadoop Admin
> >>>>> Cloudwick Technologies
> >>>>
> >>>>
> >>>
> >
>
>
>
> --
> Harsh J
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by 闫昆 <ya...@gmail.com>.

hi
what you dfs.namenode.name.dir set?
check you dfs,namenode.name.dir directory permit and make sure you user can
access


2013/7/16 Harsh J <ha...@cloudera.com>

> > 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
> This is interesting. Its calling your volume "null", which may be more
> of a superficial bug.
>
> What is your dfs.namenode.name.dir set to? From
> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
> is being used off of the out-of-box defaults. Could you try to set it
> to a specific directory thats not on /tmp?
>
> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > I don't have it in my hdfs-site.xml, in which case probably the default
> > value is taken..
> >
> >
> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
> >>
> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
> >>
> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
> wrote:
> >>>
> >>> Hi Krishna,
> >>>
> >>>    Can you please send screenshots of namenode web UI.
> >>>
> >>> Thanks Aditya.
> >>>
> >>>
> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
> >>> <wr...@gmail.com> wrote:
> >>>>
> >>>> I have had enough space on the disk that is used, like around 30 Gigs
> >>>>
> >>>> Thanks,
> >>>> Kishore
> >>>>
> >>>>
> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
> >>>> <ve...@cloudwick.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> pls see the available space for NN storage directory.
> >>>>>
> >>>>> Thanks & Regards
> >>>>>
> >>>>> Venkat
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
> >>>>> <wr...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>  I am doing no activity on my single node cluster which is using
> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
> itself after
> >>>>>> a while. I was looking at the name node log and see many of these
> kinds of
> >>>>>> entries.. Can anything be interpreted from these?
> >>>>>>
> >>>>>> 2013-07-12 09:06:11,256 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 561
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 561
> >>>>>> 2013-07-12 09:07:11,291 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
> >>>>>> 2013-07-12 09:07:11,292 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
> >>>>>> 2013-07-12 09:07:11,293 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> >>>>>> 2013-07-12 09:07:11,294 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 563
> >>>>>> 2013-07-12 09:08:11,397 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 563
> >>>>>> 2013-07-12 09:08:11,399 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
> >>>>>> 2013-07-12 09:08:11,400 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
> >>>>>> 2013-07-12 09:09:11,441 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
> >>>>>>
> >>>>>>
> >>>>>> And after sometime it said:
> >>>>>>
> >>>>>> 2013-07-12 11:03:19,799 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 795
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 795
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 797
> >>>>>> 2013-07-12 11:04:26,002 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:26,003 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Entering safe mode.
> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ON.
> >>>>>> Resources are low on NN. Please add or free up more resources then
> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
> before adding
> >>>>>> resources, the NN will immediately return to safe mode..
> >>>>>> 2013-07-12 11:04:31,004 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:31,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Already in safe mode.
> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
> up more
> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off
> safe
> >>>>>> mode before adding resources, the NN will immediately return to
> safe mode..
> >>>>>> 2013-07-12 11:04:36,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Kishore
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> N Venkata Rami Reddy
> >>>>> Hadoop Admin
> >>>>> Cloudwick Technologies
> >>>>
> >>>>
> >>>
> >
>
>
>
> --
> Harsh J
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Yes Harsh, I haven't set dfs.namenode.name.dir anywhere in config files. My
name node has again gone into safe mode today while it was idle. I shall
try setting this value to something other than /tmp



On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:

> > 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
> This is interesting. Its calling your volume "null", which may be more
> of a superficial bug.
>
> What is your dfs.namenode.name.dir set to? From
> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
> is being used off of the out-of-box defaults. Could you try to set it
> to a specific directory thats not on /tmp?
>
> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > I don't have it in my hdfs-site.xml, in which case probably the default
> > value is taken..
> >
> >
> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
> >>
> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
> >>
> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
> wrote:
> >>>
> >>> Hi Krishna,
> >>>
> >>>    Can you please send screenshots of namenode web UI.
> >>>
> >>> Thanks Aditya.
> >>>
> >>>
> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
> >>> <wr...@gmail.com> wrote:
> >>>>
> >>>> I have had enough space on the disk that is used, like around 30 Gigs
> >>>>
> >>>> Thanks,
> >>>> Kishore
> >>>>
> >>>>
> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
> >>>> <ve...@cloudwick.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> pls see the available space for NN storage directory.
> >>>>>
> >>>>> Thanks & Regards
> >>>>>
> >>>>> Venkat
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
> >>>>> <wr...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>  I am doing no activity on my single node cluster which is using
> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
> itself after
> >>>>>> a while. I was looking at the name node log and see many of these
> kinds of
> >>>>>> entries.. Can anything be interpreted from these?
> >>>>>>
> >>>>>> 2013-07-12 09:06:11,256 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 561
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 561
> >>>>>> 2013-07-12 09:07:11,291 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
> >>>>>> 2013-07-12 09:07:11,292 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
> >>>>>> 2013-07-12 09:07:11,293 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> >>>>>> 2013-07-12 09:07:11,294 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 563
> >>>>>> 2013-07-12 09:08:11,397 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 563
> >>>>>> 2013-07-12 09:08:11,399 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
> >>>>>> 2013-07-12 09:08:11,400 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
> >>>>>> 2013-07-12 09:09:11,441 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
> >>>>>>
> >>>>>>
> >>>>>> And after sometime it said:
> >>>>>>
> >>>>>> 2013-07-12 11:03:19,799 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 795
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 795
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 797
> >>>>>> 2013-07-12 11:04:26,002 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:26,003 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Entering safe mode.
> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ON.
> >>>>>> Resources are low on NN. Please add or free up more resources then
> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
> before adding
> >>>>>> resources, the NN will immediately return to safe mode..
> >>>>>> 2013-07-12 11:04:31,004 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:31,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Already in safe mode.
> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
> up more
> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off
> safe
> >>>>>> mode before adding resources, the NN will immediately return to
> safe mode..
> >>>>>> 2013-07-12 11:04:36,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Kishore
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> N Venkata Rami Reddy
> >>>>> Hadoop Admin
> >>>>> Cloudwick Technologies
> >>>>
> >>>>
> >>>
> >
>
>
>
> --
> Harsh J
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by 闫昆 <ya...@gmail.com>.

hi
what you dfs.namenode.name.dir set?
check you dfs,namenode.name.dir directory permit and make sure you user can
access


2013/7/16 Harsh J <ha...@cloudera.com>

> > 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
> This is interesting. Its calling your volume "null", which may be more
> of a superficial bug.
>
> What is your dfs.namenode.name.dir set to? From
> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
> is being used off of the out-of-box defaults. Could you try to set it
> to a specific directory thats not on /tmp?
>
> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > I don't have it in my hdfs-site.xml, in which case probably the default
> > value is taken..
> >
> >
> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
> >>
> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
> >>
> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
> wrote:
> >>>
> >>> Hi Krishna,
> >>>
> >>>    Can you please send screenshots of namenode web UI.
> >>>
> >>> Thanks Aditya.
> >>>
> >>>
> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
> >>> <wr...@gmail.com> wrote:
> >>>>
> >>>> I have had enough space on the disk that is used, like around 30 Gigs
> >>>>
> >>>> Thanks,
> >>>> Kishore
> >>>>
> >>>>
> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
> >>>> <ve...@cloudwick.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> pls see the available space for NN storage directory.
> >>>>>
> >>>>> Thanks & Regards
> >>>>>
> >>>>> Venkat
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
> >>>>> <wr...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>  I am doing no activity on my single node cluster which is using
> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
> itself after
> >>>>>> a while. I was looking at the name node log and see many of these
> kinds of
> >>>>>> entries.. Can anything be interpreted from these?
> >>>>>>
> >>>>>> 2013-07-12 09:06:11,256 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 561
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 561
> >>>>>> 2013-07-12 09:07:11,291 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
> >>>>>> 2013-07-12 09:07:11,292 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
> >>>>>> 2013-07-12 09:07:11,293 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> >>>>>> 2013-07-12 09:07:11,294 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 563
> >>>>>> 2013-07-12 09:08:11,397 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 563
> >>>>>> 2013-07-12 09:08:11,399 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
> >>>>>> 2013-07-12 09:08:11,400 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
> >>>>>> 2013-07-12 09:09:11,441 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
> >>>>>>
> >>>>>>
> >>>>>> And after sometime it said:
> >>>>>>
> >>>>>> 2013-07-12 11:03:19,799 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 795
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 795
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 797
> >>>>>> 2013-07-12 11:04:26,002 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:26,003 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Entering safe mode.
> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ON.
> >>>>>> Resources are low on NN. Please add or free up more resources then
> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
> before adding
> >>>>>> resources, the NN will immediately return to safe mode..
> >>>>>> 2013-07-12 11:04:31,004 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:31,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Already in safe mode.
> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
> up more
> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off
> safe
> >>>>>> mode before adding resources, the NN will immediately return to
> safe mode..
> >>>>>> 2013-07-12 11:04:36,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Kishore
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> N Venkata Rami Reddy
> >>>>> Hadoop Admin
> >>>>> Cloudwick Technologies
> >>>>
> >>>>
> >>>
> >
>
>
>
> --
> Harsh J
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by 闫昆 <ya...@gmail.com>.

hi
what you dfs.namenode.name.dir set?
check you dfs,namenode.name.dir directory permit and make sure you user can
access


2013/7/16 Harsh J <ha...@cloudera.com>

> > 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
> This is interesting. Its calling your volume "null", which may be more
> of a superficial bug.
>
> What is your dfs.namenode.name.dir set to? From
> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
> is being used off of the out-of-box defaults. Could you try to set it
> to a specific directory thats not on /tmp?
>
> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > I don't have it in my hdfs-site.xml, in which case probably the default
> > value is taken..
> >
> >
> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
> >>
> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
> >>
> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
> wrote:
> >>>
> >>> Hi Krishna,
> >>>
> >>>    Can you please send screenshots of namenode web UI.
> >>>
> >>> Thanks Aditya.
> >>>
> >>>
> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
> >>> <wr...@gmail.com> wrote:
> >>>>
> >>>> I have had enough space on the disk that is used, like around 30 Gigs
> >>>>
> >>>> Thanks,
> >>>> Kishore
> >>>>
> >>>>
> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
> >>>> <ve...@cloudwick.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> pls see the available space for NN storage directory.
> >>>>>
> >>>>> Thanks & Regards
> >>>>>
> >>>>> Venkat
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
> >>>>> <wr...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>  I am doing no activity on my single node cluster which is using
> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
> itself after
> >>>>>> a while. I was looking at the name node log and see many of these
> kinds of
> >>>>>> entries.. Can anything be interpreted from these?
> >>>>>>
> >>>>>> 2013-07-12 09:06:11,256 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 561
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 561
> >>>>>> 2013-07-12 09:07:11,291 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
> >>>>>> 2013-07-12 09:07:11,292 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
> >>>>>> 2013-07-12 09:07:11,293 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> >>>>>> 2013-07-12 09:07:11,294 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 563
> >>>>>> 2013-07-12 09:08:11,397 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 563
> >>>>>> 2013-07-12 09:08:11,399 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
> >>>>>> 2013-07-12 09:08:11,400 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
> >>>>>> 2013-07-12 09:09:11,441 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
> >>>>>>
> >>>>>>
> >>>>>> And after sometime it said:
> >>>>>>
> >>>>>> 2013-07-12 11:03:19,799 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 795
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 795
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 797
> >>>>>> 2013-07-12 11:04:26,002 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:26,003 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Entering safe mode.
> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ON.
> >>>>>> Resources are low on NN. Please add or free up more resources then
> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
> before adding
> >>>>>> resources, the NN will immediately return to safe mode..
> >>>>>> 2013-07-12 11:04:31,004 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:31,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Already in safe mode.
> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
> up more
> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off
> safe
> >>>>>> mode before adding resources, the NN will immediately return to
> safe mode..
> >>>>>> 2013-07-12 11:04:36,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Kishore
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> N Venkata Rami Reddy
> >>>>> Hadoop Admin
> >>>>> Cloudwick Technologies
> >>>>
> >>>>
> >>>
> >
>
>
>
> --
> Harsh J
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by 闫昆 <ya...@gmail.com>.

hi
what you dfs.namenode.name.dir set?
check you dfs,namenode.name.dir directory permit and make sure you user can
access


2013/7/16 Harsh J <ha...@cloudera.com>

> > 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
> This is interesting. Its calling your volume "null", which may be more
> of a superficial bug.
>
> What is your dfs.namenode.name.dir set to? From
> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
> is being used off of the out-of-box defaults. Could you try to set it
> to a specific directory thats not on /tmp?
>
> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > I don't have it in my hdfs-site.xml, in which case probably the default
> > value is taken..
> >
> >
> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
> >>
> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
> >>
> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
> wrote:
> >>>
> >>> Hi Krishna,
> >>>
> >>>    Can you please send screenshots of namenode web UI.
> >>>
> >>> Thanks Aditya.
> >>>
> >>>
> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
> >>> <wr...@gmail.com> wrote:
> >>>>
> >>>> I have had enough space on the disk that is used, like around 30 Gigs
> >>>>
> >>>> Thanks,
> >>>> Kishore
> >>>>
> >>>>
> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
> >>>> <ve...@cloudwick.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> pls see the available space for NN storage directory.
> >>>>>
> >>>>> Thanks & Regards
> >>>>>
> >>>>> Venkat
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
> >>>>> <wr...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>  I am doing no activity on my single node cluster which is using
> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
> itself after
> >>>>>> a while. I was looking at the name node log and see many of these
> kinds of
> >>>>>> entries.. Can anything be interpreted from these?
> >>>>>>
> >>>>>> 2013-07-12 09:06:11,256 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 561
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 561
> >>>>>> 2013-07-12 09:07:11,291 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
> >>>>>> 2013-07-12 09:07:11,292 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
> >>>>>> 2013-07-12 09:07:11,293 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> >>>>>> 2013-07-12 09:07:11,294 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 563
> >>>>>> 2013-07-12 09:08:11,397 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 563
> >>>>>> 2013-07-12 09:08:11,399 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
> >>>>>> 2013-07-12 09:08:11,400 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
> >>>>>> 2013-07-12 09:09:11,441 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
> >>>>>>
> >>>>>>
> >>>>>> And after sometime it said:
> >>>>>>
> >>>>>> 2013-07-12 11:03:19,799 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 795
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 795
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 797
> >>>>>> 2013-07-12 11:04:26,002 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:26,003 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Entering safe mode.
> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ON.
> >>>>>> Resources are low on NN. Please add or free up more resources then
> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
> before adding
> >>>>>> resources, the NN will immediately return to safe mode..
> >>>>>> 2013-07-12 11:04:31,004 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:31,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Already in safe mode.
> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
> up more
> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off
> safe
> >>>>>> mode before adding resources, the NN will immediately return to
> safe mode..
> >>>>>> 2013-07-12 11:04:36,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Kishore
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> N Venkata Rami Reddy
> >>>>> Hadoop Admin
> >>>>> Cloudwick Technologies
> >>>>
> >>>>
> >>>
> >
>
>
>
> --
> Harsh J
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Yes Harsh, I haven't set dfs.namenode.name.dir anywhere in config files. My
name node has again gone into safe mode today while it was idle. I shall
try setting this value to something other than /tmp



On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:

> > 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
> This is interesting. Its calling your volume "null", which may be more
> of a superficial bug.
>
> What is your dfs.namenode.name.dir set to? From
> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
> is being used off of the out-of-box defaults. Could you try to set it
> to a specific directory thats not on /tmp?
>
> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > I don't have it in my hdfs-site.xml, in which case probably the default
> > value is taken..
> >
> >
> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
> >>
> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
> >>
> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
> wrote:
> >>>
> >>> Hi Krishna,
> >>>
> >>>    Can you please send screenshots of namenode web UI.
> >>>
> >>> Thanks Aditya.
> >>>
> >>>
> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
> >>> <wr...@gmail.com> wrote:
> >>>>
> >>>> I have had enough space on the disk that is used, like around 30 Gigs
> >>>>
> >>>> Thanks,
> >>>> Kishore
> >>>>
> >>>>
> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
> >>>> <ve...@cloudwick.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> pls see the available space for NN storage directory.
> >>>>>
> >>>>> Thanks & Regards
> >>>>>
> >>>>> Venkat
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
> >>>>> <wr...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>  I am doing no activity on my single node cluster which is using
> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
> itself after
> >>>>>> a while. I was looking at the name node log and see many of these
> kinds of
> >>>>>> entries.. Can anything be interpreted from these?
> >>>>>>
> >>>>>> 2013-07-12 09:06:11,256 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 561
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 561
> >>>>>> 2013-07-12 09:07:11,291 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
> >>>>>> 2013-07-12 09:07:11,292 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
> >>>>>> 2013-07-12 09:07:11,293 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> >>>>>> 2013-07-12 09:07:11,294 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 563
> >>>>>> 2013-07-12 09:08:11,397 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 563
> >>>>>> 2013-07-12 09:08:11,399 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
> >>>>>> 2013-07-12 09:08:11,400 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
> >>>>>> 2013-07-12 09:09:11,441 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
> >>>>>>
> >>>>>>
> >>>>>> And after sometime it said:
> >>>>>>
> >>>>>> 2013-07-12 11:03:19,799 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 795
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 795
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 797
> >>>>>> 2013-07-12 11:04:26,002 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:26,003 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Entering safe mode.
> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ON.
> >>>>>> Resources are low on NN. Please add or free up more resources then
> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
> before adding
> >>>>>> resources, the NN will immediately return to safe mode..
> >>>>>> 2013-07-12 11:04:31,004 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:31,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Already in safe mode.
> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
> up more
> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off
> safe
> >>>>>> mode before adding resources, the NN will immediately return to
> safe mode..
> >>>>>> 2013-07-12 11:04:36,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Kishore
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> N Venkata Rami Reddy
> >>>>> Hadoop Admin
> >>>>> Cloudwick Technologies
> >>>>
> >>>>
> >>>
> >
>
>
>
> --
> Harsh J
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Yes Harsh, I haven't set dfs.namenode.name.dir anywhere in config files. My
name node has again gone into safe mode today while it was idle. I shall
try setting this value to something other than /tmp



On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:

> > 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
> This is interesting. Its calling your volume "null", which may be more
> of a superficial bug.
>
> What is your dfs.namenode.name.dir set to? From
> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
> is being used off of the out-of-box defaults. Could you try to set it
> to a specific directory thats not on /tmp?
>
> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > I don't have it in my hdfs-site.xml, in which case probably the default
> > value is taken..
> >
> >
> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
> >>
> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
> >>
> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
> wrote:
> >>>
> >>> Hi Krishna,
> >>>
> >>>    Can you please send screenshots of namenode web UI.
> >>>
> >>> Thanks Aditya.
> >>>
> >>>
> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
> >>> <wr...@gmail.com> wrote:
> >>>>
> >>>> I have had enough space on the disk that is used, like around 30 Gigs
> >>>>
> >>>> Thanks,
> >>>> Kishore
> >>>>
> >>>>
> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
> >>>> <ve...@cloudwick.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> pls see the available space for NN storage directory.
> >>>>>
> >>>>> Thanks & Regards
> >>>>>
> >>>>> Venkat
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
> >>>>> <wr...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>  I am doing no activity on my single node cluster which is using
> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
> itself after
> >>>>>> a while. I was looking at the name node log and see many of these
> kinds of
> >>>>>> entries.. Can anything be interpreted from these?
> >>>>>>
> >>>>>> 2013-07-12 09:06:11,256 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 561
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 561
> >>>>>> 2013-07-12 09:07:11,291 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
> >>>>>> 2013-07-12 09:07:11,292 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
> >>>>>> 2013-07-12 09:07:11,293 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> >>>>>> 2013-07-12 09:07:11,294 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 563
> >>>>>> 2013-07-12 09:08:11,397 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 563
> >>>>>> 2013-07-12 09:08:11,399 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
> >>>>>> 2013-07-12 09:08:11,400 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
> >>>>>> 2013-07-12 09:09:11,441 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
> >>>>>>
> >>>>>>
> >>>>>> And after sometime it said:
> >>>>>>
> >>>>>> 2013-07-12 11:03:19,799 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 795
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 795
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 797
> >>>>>> 2013-07-12 11:04:26,002 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:26,003 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Entering safe mode.
> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ON.
> >>>>>> Resources are low on NN. Please add or free up more resources then
> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
> before adding
> >>>>>> resources, the NN will immediately return to safe mode..
> >>>>>> 2013-07-12 11:04:31,004 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:31,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Already in safe mode.
> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
> up more
> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off
> safe
> >>>>>> mode before adding resources, the NN will immediately return to
> safe mode..
> >>>>>> 2013-07-12 11:04:36,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Kishore
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> N Venkata Rami Reddy
> >>>>> Hadoop Admin
> >>>>> Cloudwick Technologies
> >>>>
> >>>>
> >>>
> >
>
>
>
> --
> Harsh J
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Harsh,

  I have made my dfs.namenode.name.dir point to a subdirectory of my home,
and I don't see this issue again. So, is this a bug that we need to log
into JIRA?

Thanks,
Kishore


On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:

> > 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
> This is interesting. Its calling your volume "null", which may be more
> of a superficial bug.
>
> What is your dfs.namenode.name.dir set to? From
> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
> is being used off of the out-of-box defaults. Could you try to set it
> to a specific directory thats not on /tmp?
>
> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > I don't have it in my hdfs-site.xml, in which case probably the default
> > value is taken..
> >
> >
> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
> >>
> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
> >>
> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
> wrote:
> >>>
> >>> Hi Krishna,
> >>>
> >>>    Can you please send screenshots of namenode web UI.
> >>>
> >>> Thanks Aditya.
> >>>
> >>>
> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
> >>> <wr...@gmail.com> wrote:
> >>>>
> >>>> I have had enough space on the disk that is used, like around 30 Gigs
> >>>>
> >>>> Thanks,
> >>>> Kishore
> >>>>
> >>>>
> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
> >>>> <ve...@cloudwick.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> pls see the available space for NN storage directory.
> >>>>>
> >>>>> Thanks & Regards
> >>>>>
> >>>>> Venkat
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
> >>>>> <wr...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>  I am doing no activity on my single node cluster which is using
> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
> itself after
> >>>>>> a while. I was looking at the name node log and see many of these
> kinds of
> >>>>>> entries.. Can anything be interpreted from these?
> >>>>>>
> >>>>>> 2013-07-12 09:06:11,256 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 561
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 561
> >>>>>> 2013-07-12 09:07:11,291 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
> >>>>>> 2013-07-12 09:07:11,292 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
> >>>>>> 2013-07-12 09:07:11,293 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> >>>>>> 2013-07-12 09:07:11,294 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 563
> >>>>>> 2013-07-12 09:08:11,397 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 563
> >>>>>> 2013-07-12 09:08:11,399 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
> >>>>>> 2013-07-12 09:08:11,400 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
> >>>>>> 2013-07-12 09:09:11,441 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
> >>>>>>
> >>>>>>
> >>>>>> And after sometime it said:
> >>>>>>
> >>>>>> 2013-07-12 11:03:19,799 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 795
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 795
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 797
> >>>>>> 2013-07-12 11:04:26,002 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:26,003 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Entering safe mode.
> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ON.
> >>>>>> Resources are low on NN. Please add or free up more resources then
> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
> before adding
> >>>>>> resources, the NN will immediately return to safe mode..
> >>>>>> 2013-07-12 11:04:31,004 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:31,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Already in safe mode.
> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
> up more
> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off
> safe
> >>>>>> mode before adding resources, the NN will immediately return to
> safe mode..
> >>>>>> 2013-07-12 11:04:36,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Kishore
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> N Venkata Rami Reddy
> >>>>> Hadoop Admin
> >>>>> Cloudwick Technologies
> >>>>
> >>>>
> >>>
> >
>
>
>
> --
> Harsh J
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Harsh,

  I have made my dfs.namenode.name.dir point to a subdirectory of my home,
and I don't see this issue again. So, is this a bug that we need to log
into JIRA?

Thanks,
Kishore


On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote:

> > 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
> This is interesting. Its calling your volume "null", which may be more
> of a superficial bug.
>
> What is your dfs.namenode.name.dir set to? From
> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
> is being used off of the out-of-box defaults. Could you try to set it
> to a specific directory thats not on /tmp?
>
> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > I don't have it in my hdfs-site.xml, in which case probably the default
> > value is taken..
> >
> >
> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
> >>
> >> please check dfs.datanode.du.reserved in the hdfs-site.xml
> >>
> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com>
> wrote:
> >>>
> >>> Hi Krishna,
> >>>
> >>>    Can you please send screenshots of namenode web UI.
> >>>
> >>> Thanks Aditya.
> >>>
> >>>
> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
> >>> <wr...@gmail.com> wrote:
> >>>>
> >>>> I have had enough space on the disk that is used, like around 30 Gigs
> >>>>
> >>>> Thanks,
> >>>> Kishore
> >>>>
> >>>>
> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
> >>>> <ve...@cloudwick.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> pls see the available space for NN storage directory.
> >>>>>
> >>>>> Thanks & Regards
> >>>>>
> >>>>> Venkat
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
> >>>>> <wr...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>  I am doing no activity on my single node cluster which is using
> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by
> itself after
> >>>>>> a while. I was looking at the name node log and see many of these
> kinds of
> >>>>>> entries.. Can anything be interpreted from these?
> >>>>>>
> >>>>>> 2013-07-12 09:06:11,256 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 561
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:07:11,290 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 561
> >>>>>> 2013-07-12 09:07:11,291 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
> >>>>>> 2013-07-12 09:07:11,292 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 1 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
> >>>>>> 2013-07-12 09:07:11,293 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> >>>>>> 2013-07-12 09:07:11,294 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 563
> >>>>>> 2013-07-12 09:08:11,397 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:08:11,398 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 563
> >>>>>> 2013-07-12 09:08:11,399 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
> >>>>>> 2013-07-12 09:08:11,400 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 2 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> >>>>>> 2013-07-12 09:08:11,402 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 565
> >>>>>> 2013-07-12 09:09:11,440 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
> >>>>>> 2013-07-12 09:09:11,441 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
> >>>>>>
> >>>>>>
> >>>>>> And after sometime it said:
> >>>>>>
> >>>>>> 2013-07-12 11:03:19,799 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 795
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from
> >>>>>> 9.70.137.114
> >>>>>> 2013-07-12 11:04:19,826 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log
> segment 795
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,827 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
> transactions: 2
> >>>>>> Total time for transactions(ms): 0 Number of transactions batched
> in Syncs:
> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager:
> Finalizing edits
> >>>>>> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
> >>>>>> ->
> >>>>>>
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> >>>>>> 2013-07-12 11:04:19,829 INFO
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log
> segment at
> >>>>>> 797
> >>>>>> 2013-07-12 11:04:26,002 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:26,003 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Entering safe mode.
> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ON.
> >>>>>> Resources are low on NN. Please add or free up more resources then
> >>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode
> before adding
> >>>>>> resources, the NN will immediately return to safe mode..
> >>>>>> 2013-07-12 11:04:31,004 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>> 2013-07-12 11:04:31,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> >>>>>> available disk space. Already in safe mode.
> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free
> up more
> >>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off
> safe
> >>>>>> mode before adding resources, the NN will immediately return to
> safe mode..
> >>>>>> 2013-07-12 11:04:36,005 WARN
> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker:
> Space
> >>>>>> available on volume 'null' is 0, which is below the configured
> reserved
> >>>>>> amount 104857600
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Kishore
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> N Venkata Rami Reddy
> >>>>> Hadoop Admin
> >>>>> Cloudwick Technologies
> >>>>
> >>>>
> >>>
> >
>
>
>
> --
> Harsh J
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Harsh J <ha...@cloudera.com>.

> 2013-07-12 11:04:26,002 WARN org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space available on volume 'null' is 0, which is below the configured reserved amount 104857600

This is interesting. Its calling your volume "null", which may be more
of a superficial bug.

What is your dfs.namenode.name.dir set to? From
/tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
is being used off of the out-of-box defaults. Could you try to set it
to a specific directory thats not on /tmp?

On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
<wr...@gmail.com> wrote:
> I don't have it in my hdfs-site.xml, in which case probably the default
> value is taken..
>
>
> On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
>>
>> please check dfs.datanode.du.reserved in the hdfs-site.xml
>>
>> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com> wrote:
>>>
>>> Hi Krishna,
>>>
>>>    Can you please send screenshots of namenode web UI.
>>>
>>> Thanks Aditya.
>>>
>>>
>>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
>>> <wr...@gmail.com> wrote:
>>>>
>>>> I have had enough space on the disk that is used, like around 30 Gigs
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
>>>> <ve...@cloudwick.com> wrote:
>>>>>
>>>>> Hi,
>>>>> pls see the available space for NN storage directory.
>>>>>
>>>>> Thanks & Regards
>>>>>
>>>>> Venkat
>>>>>
>>>>>
>>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
>>>>> <wr...@gmail.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>  I am doing no activity on my single node cluster which is using
>>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself after
>>>>>> a while. I was looking at the name node log and see many of these kinds of
>>>>>> entries.. Can anything be interpreted from these?
>>>>>>
>>>>>> 2013-07-12 09:06:11,256 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 561
>>>>>> 2013-07-12 09:07:11,290 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 09:07:11,290 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 09:07:11,290 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>>>> 2013-07-12 09:07:11,291 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>>>> 2013-07-12 09:07:11,292 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>>>> 2013-07-12 09:07:11,293 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>>> file /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
>>>>>> ->
>>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>>>> 2013-07-12 09:07:11,294 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 563
>>>>>> 2013-07-12 09:08:11,397 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 09:08:11,398 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 09:08:11,398 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>>>> 2013-07-12 09:08:11,399 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>>>> 2013-07-12 09:08:11,400 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>>> 2013-07-12 09:08:11,402 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>>> file /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
>>>>>> ->
>>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>>>> 2013-07-12 09:08:11,402 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 565
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>>>> 2013-07-12 09:09:11,441 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>>>
>>>>>>
>>>>>> And after sometime it said:
>>>>>>
>>>>>> 2013-07-12 11:03:19,799 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 795
>>>>>> 2013-07-12 11:04:19,826 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 11:04:19,826 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 11:04:19,827 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>>>> 2013-07-12 11:04:19,827 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>>>> 2013-07-12 11:04:19,827 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>>> 2013-07-12 11:04:19,829 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>>> file /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
>>>>>> ->
>>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>>>> 2013-07-12 11:04:19,829 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 797
>>>>>> 2013-07-12 11:04:26,002 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>>> amount 104857600
>>>>>> 2013-07-12 11:04:26,003 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>>> available disk space. Entering safe mode.
>>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>>>>>> STATE* Safe mode is ON.
>>>>>> Resources are low on NN. Please add or free up more resources then
>>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>>>>> resources, the NN will immediately return to safe mode..
>>>>>> 2013-07-12 11:04:31,004 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>>> amount 104857600
>>>>>> 2013-07-12 11:04:31,005 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>>> available disk space. Already in safe mode.
>>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free up more
>>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>>>> 2013-07-12 11:04:36,005 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>>> amount 104857600
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> N Venkata Rami Reddy
>>>>> Hadoop Admin
>>>>> Cloudwick Technologies
>>>>
>>>>
>>>
>



-- 
Harsh J

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Harsh J <ha...@cloudera.com>.

> 2013-07-12 11:04:26,002 WARN org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space available on volume 'null' is 0, which is below the configured reserved amount 104857600

This is interesting. Its calling your volume "null", which may be more
of a superficial bug.

What is your dfs.namenode.name.dir set to? From
/tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
is being used off of the out-of-box defaults. Could you try to set it
to a specific directory thats not on /tmp?

On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
<wr...@gmail.com> wrote:
> I don't have it in my hdfs-site.xml, in which case probably the default
> value is taken..
>
>
> On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
>>
>> please check dfs.datanode.du.reserved in the hdfs-site.xml
>>
>> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com> wrote:
>>>
>>> Hi Krishna,
>>>
>>>    Can you please send screenshots of namenode web UI.
>>>
>>> Thanks Aditya.
>>>
>>>
>>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
>>> <wr...@gmail.com> wrote:
>>>>
>>>> I have had enough space on the disk that is used, like around 30 Gigs
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
>>>> <ve...@cloudwick.com> wrote:
>>>>>
>>>>> Hi,
>>>>> pls see the available space for NN storage directory.
>>>>>
>>>>> Thanks & Regards
>>>>>
>>>>> Venkat
>>>>>
>>>>>
>>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
>>>>> <wr...@gmail.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>  I am doing no activity on my single node cluster which is using
>>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself after
>>>>>> a while. I was looking at the name node log and see many of these kinds of
>>>>>> entries.. Can anything be interpreted from these?
>>>>>>
>>>>>> 2013-07-12 09:06:11,256 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 561
>>>>>> 2013-07-12 09:07:11,290 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 09:07:11,290 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 09:07:11,290 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>>>> 2013-07-12 09:07:11,291 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>>>> 2013-07-12 09:07:11,292 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>>>> 2013-07-12 09:07:11,293 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>>> file /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
>>>>>> ->
>>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>>>> 2013-07-12 09:07:11,294 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 563
>>>>>> 2013-07-12 09:08:11,397 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 09:08:11,398 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 09:08:11,398 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>>>> 2013-07-12 09:08:11,399 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>>>> 2013-07-12 09:08:11,400 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>>> 2013-07-12 09:08:11,402 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>>> file /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
>>>>>> ->
>>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>>>> 2013-07-12 09:08:11,402 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 565
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>>>> 2013-07-12 09:09:11,441 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>>>
>>>>>>
>>>>>> And after sometime it said:
>>>>>>
>>>>>> 2013-07-12 11:03:19,799 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 795
>>>>>> 2013-07-12 11:04:19,826 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 11:04:19,826 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 11:04:19,827 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>>>> 2013-07-12 11:04:19,827 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>>>> 2013-07-12 11:04:19,827 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>>> 2013-07-12 11:04:19,829 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>>> file /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
>>>>>> ->
>>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>>>> 2013-07-12 11:04:19,829 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 797
>>>>>> 2013-07-12 11:04:26,002 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>>> amount 104857600
>>>>>> 2013-07-12 11:04:26,003 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>>> available disk space. Entering safe mode.
>>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>>>>>> STATE* Safe mode is ON.
>>>>>> Resources are low on NN. Please add or free up more resources then
>>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>>>>> resources, the NN will immediately return to safe mode..
>>>>>> 2013-07-12 11:04:31,004 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>>> amount 104857600
>>>>>> 2013-07-12 11:04:31,005 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>>> available disk space. Already in safe mode.
>>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free up more
>>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>>>> 2013-07-12 11:04:36,005 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>>> amount 104857600
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> N Venkata Rami Reddy
>>>>> Hadoop Admin
>>>>> Cloudwick Technologies
>>>>
>>>>
>>>
>



-- 
Harsh J

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Harsh J <ha...@cloudera.com>.

> 2013-07-12 11:04:26,002 WARN org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space available on volume 'null' is 0, which is below the configured reserved amount 104857600

This is interesting. Its calling your volume "null", which may be more
of a superficial bug.

What is your dfs.namenode.name.dir set to? From
/tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
is being used off of the out-of-box defaults. Could you try to set it
to a specific directory thats not on /tmp?

On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
<wr...@gmail.com> wrote:
> I don't have it in my hdfs-site.xml, in which case probably the default
> value is taken..
>
>
> On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
>>
>> please check dfs.datanode.du.reserved in the hdfs-site.xml
>>
>> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com> wrote:
>>>
>>> Hi Krishna,
>>>
>>>    Can you please send screenshots of namenode web UI.
>>>
>>> Thanks Aditya.
>>>
>>>
>>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
>>> <wr...@gmail.com> wrote:
>>>>
>>>> I have had enough space on the disk that is used, like around 30 Gigs
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
>>>> <ve...@cloudwick.com> wrote:
>>>>>
>>>>> Hi,
>>>>> pls see the available space for NN storage directory.
>>>>>
>>>>> Thanks & Regards
>>>>>
>>>>> Venkat
>>>>>
>>>>>
>>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
>>>>> <wr...@gmail.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>  I am doing no activity on my single node cluster which is using
>>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself after
>>>>>> a while. I was looking at the name node log and see many of these kinds of
>>>>>> entries.. Can anything be interpreted from these?
>>>>>>
>>>>>> 2013-07-12 09:06:11,256 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 561
>>>>>> 2013-07-12 09:07:11,290 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 09:07:11,290 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 09:07:11,290 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>>>> 2013-07-12 09:07:11,291 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>>>> 2013-07-12 09:07:11,292 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>>>> 2013-07-12 09:07:11,293 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>>> file /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
>>>>>> ->
>>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>>>> 2013-07-12 09:07:11,294 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 563
>>>>>> 2013-07-12 09:08:11,397 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 09:08:11,398 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 09:08:11,398 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>>>> 2013-07-12 09:08:11,399 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>>>> 2013-07-12 09:08:11,400 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>>> 2013-07-12 09:08:11,402 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>>> file /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
>>>>>> ->
>>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>>>> 2013-07-12 09:08:11,402 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 565
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>>>> 2013-07-12 09:09:11,441 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>>>
>>>>>>
>>>>>> And after sometime it said:
>>>>>>
>>>>>> 2013-07-12 11:03:19,799 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 795
>>>>>> 2013-07-12 11:04:19,826 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 11:04:19,826 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 11:04:19,827 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>>>> 2013-07-12 11:04:19,827 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>>>> 2013-07-12 11:04:19,827 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>>> 2013-07-12 11:04:19,829 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>>> file /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
>>>>>> ->
>>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>>>> 2013-07-12 11:04:19,829 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 797
>>>>>> 2013-07-12 11:04:26,002 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>>> amount 104857600
>>>>>> 2013-07-12 11:04:26,003 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>>> available disk space. Entering safe mode.
>>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>>>>>> STATE* Safe mode is ON.
>>>>>> Resources are low on NN. Please add or free up more resources then
>>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>>>>> resources, the NN will immediately return to safe mode..
>>>>>> 2013-07-12 11:04:31,004 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>>> amount 104857600
>>>>>> 2013-07-12 11:04:31,005 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>>> available disk space. Already in safe mode.
>>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free up more
>>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>>>> 2013-07-12 11:04:36,005 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>>> amount 104857600
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> N Venkata Rami Reddy
>>>>> Hadoop Admin
>>>>> Cloudwick Technologies
>>>>
>>>>
>>>
>



-- 
Harsh J

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Harsh J <ha...@cloudera.com>.

> 2013-07-12 11:04:26,002 WARN org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space available on volume 'null' is 0, which is below the configured reserved amount 104857600

This is interesting. Its calling your volume "null", which may be more
of a superficial bug.

What is your dfs.namenode.name.dir set to? From
/tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp
is being used off of the out-of-box defaults. Could you try to set it
to a specific directory thats not on /tmp?

On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri
<wr...@gmail.com> wrote:
> I don't have it in my hdfs-site.xml, in which case probably the default
> value is taken..
>
>
> On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:
>>
>> please check dfs.datanode.du.reserved in the hdfs-site.xml
>>
>> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com> wrote:
>>>
>>> Hi Krishna,
>>>
>>>    Can you please send screenshots of namenode web UI.
>>>
>>> Thanks Aditya.
>>>
>>>
>>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri
>>> <wr...@gmail.com> wrote:
>>>>
>>>> I have had enough space on the disk that is used, like around 30 Gigs
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla
>>>> <ve...@cloudwick.com> wrote:
>>>>>
>>>>> Hi,
>>>>> pls see the available space for NN storage directory.
>>>>>
>>>>> Thanks & Regards
>>>>>
>>>>> Venkat
>>>>>
>>>>>
>>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri
>>>>> <wr...@gmail.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>  I am doing no activity on my single node cluster which is using
>>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself after
>>>>>> a while. I was looking at the name node log and see many of these kinds of
>>>>>> entries.. Can anything be interpreted from these?
>>>>>>
>>>>>> 2013-07-12 09:06:11,256 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 561
>>>>>> 2013-07-12 09:07:11,290 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 09:07:11,290 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 09:07:11,290 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>>>> 2013-07-12 09:07:11,291 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>>>> 2013-07-12 09:07:11,292 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>>>> 2013-07-12 09:07:11,293 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>>> file /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561
>>>>>> ->
>>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>>>> 2013-07-12 09:07:11,294 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 563
>>>>>> 2013-07-12 09:08:11,397 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 09:08:11,398 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 09:08:11,398 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>>>> 2013-07-12 09:08:11,399 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>>>> 2013-07-12 09:08:11,400 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>>> 2013-07-12 09:08:11,402 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>>> file /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563
>>>>>> ->
>>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>>>> 2013-07-12 09:08:11,402 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 565
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>>>> 2013-07-12 09:09:11,440 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>>>> 2013-07-12 09:09:11,441 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>>>
>>>>>>
>>>>>> And after sometime it said:
>>>>>>
>>>>>> 2013-07-12 11:03:19,799 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 795
>>>>>> 2013-07-12 11:04:19,826 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>>> 9.70.137.114
>>>>>> 2013-07-12 11:04:19,826 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>>> 2013-07-12 11:04:19,827 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>>>> 2013-07-12 11:04:19,827 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>>>> 2013-07-12 11:04:19,827 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>>> 2013-07-12 11:04:19,829 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>>> file /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795
>>>>>> ->
>>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>>>> 2013-07-12 11:04:19,829 INFO
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>>> 797
>>>>>> 2013-07-12 11:04:26,002 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>>> amount 104857600
>>>>>> 2013-07-12 11:04:26,003 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>>> available disk space. Entering safe mode.
>>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>>>>>> STATE* Safe mode is ON.
>>>>>> Resources are low on NN. Please add or free up more resources then
>>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>>>>> resources, the NN will immediately return to safe mode..
>>>>>> 2013-07-12 11:04:31,004 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>>> amount 104857600
>>>>>> 2013-07-12 11:04:31,005 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>>> available disk space. Already in safe mode.
>>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free up more
>>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>>>> 2013-07-12 11:04:36,005 WARN
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>>> amount 104857600
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> N Venkata Rami Reddy
>>>>> Hadoop Admin
>>>>> Cloudwick Technologies
>>>>
>>>>
>>>
>



-- 
Harsh J

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

I don't have it in my hdfs-site.xml, in which case probably the default
value is taken..


On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:

> please check dfs.datanode.du.reserved in the hdfs-site.xml
> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com> wrote:
>
>> Hi Krishna,
>>
>>    Can you please send screenshots of namenode web UI.
>>
>> Thanks Aditya.
>>
>>
>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> I have had enough space on the disk that is used, like around 30 Gigs
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
>>> venkatarami.netla@cloudwick.com> wrote:
>>>
>>>> Hi,
>>>> pls see the available space for NN storage directory.
>>>>
>>>> Thanks & Regards
>>>>
>>>> Venkat
>>>>
>>>>
>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>>>> write2kishore@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>>  I am doing no activity on my single node cluster which is using
>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>>>> after a while. I was looking at the name node log and see many of these
>>>>> kinds of entries.. Can anything be interpreted from these?
>>>>>
>>>>> 2013-07-12 09:06:11,256 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 561
>>>>> 2013-07-12 09:07:11,290 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 09:07:11,290 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 09:07:11,290 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>>> 2013-07-12 09:07:11,291 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>>> 2013-07-12 09:07:11,292 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>>> 2013-07-12 09:07:11,293 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>> file
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>>> 2013-07-12 09:07:11,294 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 563
>>>>> 2013-07-12 09:08:11,397 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 09:08:11,398 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 09:08:11,398 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>>> 2013-07-12 09:08:11,399 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>>> 2013-07-12 09:08:11,400 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>> 2013-07-12 09:08:11,402 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>> file
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>>> 2013-07-12 09:08:11,402 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 565
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>>> 2013-07-12 09:09:11,441 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>>
>>>>>
>>>>> And after sometime it said:
>>>>>
>>>>> 2013-07-12 11:03:19,799 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 795
>>>>> 2013-07-12 11:04:19,826 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 11:04:19,826 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 11:04:19,827 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>>> 2013-07-12 11:04:19,827 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>>> 2013-07-12 11:04:19,827 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>> 2013-07-12 11:04:19,829 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>> file
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>>> 2013-07-12 11:04:19,829 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 797
>>>>> 2013-07-12 11:04:26,002 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>> amount 104857600
>>>>> 2013-07-12 11:04:26,003 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>> available disk space. Entering safe mode.
>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>>>>> STATE* Safe mode is ON.
>>>>> Resources are low on NN. Please add or free up more resources then
>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode before
>>>>> adding resources, the NN will immediately return to safe mode..
>>>>> 2013-07-12 11:04:31,004 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>> amount 104857600
>>>>> 2013-07-12 11:04:31,005 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>> available disk space. Already in safe mode.
>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free up more
>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>>> 2013-07-12 11:04:36,005 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>> amount 104857600
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> N Venkata Rami Reddy
>>>> Hadoop Admin
>>>> Cloudwick Technologies
>>>>
>>>
>>>
>>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

I don't have it in my hdfs-site.xml, in which case probably the default
value is taken..


On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:

> please check dfs.datanode.du.reserved in the hdfs-site.xml
> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com> wrote:
>
>> Hi Krishna,
>>
>>    Can you please send screenshots of namenode web UI.
>>
>> Thanks Aditya.
>>
>>
>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> I have had enough space on the disk that is used, like around 30 Gigs
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
>>> venkatarami.netla@cloudwick.com> wrote:
>>>
>>>> Hi,
>>>> pls see the available space for NN storage directory.
>>>>
>>>> Thanks & Regards
>>>>
>>>> Venkat
>>>>
>>>>
>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>>>> write2kishore@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>>  I am doing no activity on my single node cluster which is using
>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>>>> after a while. I was looking at the name node log and see many of these
>>>>> kinds of entries.. Can anything be interpreted from these?
>>>>>
>>>>> 2013-07-12 09:06:11,256 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 561
>>>>> 2013-07-12 09:07:11,290 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 09:07:11,290 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 09:07:11,290 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>>> 2013-07-12 09:07:11,291 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>>> 2013-07-12 09:07:11,292 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>>> 2013-07-12 09:07:11,293 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>> file
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>>> 2013-07-12 09:07:11,294 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 563
>>>>> 2013-07-12 09:08:11,397 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 09:08:11,398 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 09:08:11,398 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>>> 2013-07-12 09:08:11,399 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>>> 2013-07-12 09:08:11,400 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>> 2013-07-12 09:08:11,402 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>> file
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>>> 2013-07-12 09:08:11,402 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 565
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>>> 2013-07-12 09:09:11,441 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>>
>>>>>
>>>>> And after sometime it said:
>>>>>
>>>>> 2013-07-12 11:03:19,799 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 795
>>>>> 2013-07-12 11:04:19,826 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 11:04:19,826 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 11:04:19,827 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>>> 2013-07-12 11:04:19,827 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>>> 2013-07-12 11:04:19,827 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>> 2013-07-12 11:04:19,829 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>> file
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>>> 2013-07-12 11:04:19,829 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 797
>>>>> 2013-07-12 11:04:26,002 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>> amount 104857600
>>>>> 2013-07-12 11:04:26,003 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>> available disk space. Entering safe mode.
>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>>>>> STATE* Safe mode is ON.
>>>>> Resources are low on NN. Please add or free up more resources then
>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode before
>>>>> adding resources, the NN will immediately return to safe mode..
>>>>> 2013-07-12 11:04:31,004 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>> amount 104857600
>>>>> 2013-07-12 11:04:31,005 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>> available disk space. Already in safe mode.
>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free up more
>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>>> 2013-07-12 11:04:36,005 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>> amount 104857600
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> N Venkata Rami Reddy
>>>> Hadoop Admin
>>>> Cloudwick Technologies
>>>>
>>>
>>>
>>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

I don't have it in my hdfs-site.xml, in which case probably the default
value is taken..


On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:

> please check dfs.datanode.du.reserved in the hdfs-site.xml
> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com> wrote:
>
>> Hi Krishna,
>>
>>    Can you please send screenshots of namenode web UI.
>>
>> Thanks Aditya.
>>
>>
>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> I have had enough space on the disk that is used, like around 30 Gigs
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
>>> venkatarami.netla@cloudwick.com> wrote:
>>>
>>>> Hi,
>>>> pls see the available space for NN storage directory.
>>>>
>>>> Thanks & Regards
>>>>
>>>> Venkat
>>>>
>>>>
>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>>>> write2kishore@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>>  I am doing no activity on my single node cluster which is using
>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>>>> after a while. I was looking at the name node log and see many of these
>>>>> kinds of entries.. Can anything be interpreted from these?
>>>>>
>>>>> 2013-07-12 09:06:11,256 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 561
>>>>> 2013-07-12 09:07:11,290 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 09:07:11,290 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 09:07:11,290 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>>> 2013-07-12 09:07:11,291 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>>> 2013-07-12 09:07:11,292 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>>> 2013-07-12 09:07:11,293 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>> file
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>>> 2013-07-12 09:07:11,294 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 563
>>>>> 2013-07-12 09:08:11,397 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 09:08:11,398 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 09:08:11,398 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>>> 2013-07-12 09:08:11,399 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>>> 2013-07-12 09:08:11,400 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>> 2013-07-12 09:08:11,402 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>> file
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>>> 2013-07-12 09:08:11,402 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 565
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>>> 2013-07-12 09:09:11,441 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>>
>>>>>
>>>>> And after sometime it said:
>>>>>
>>>>> 2013-07-12 11:03:19,799 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 795
>>>>> 2013-07-12 11:04:19,826 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 11:04:19,826 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 11:04:19,827 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>>> 2013-07-12 11:04:19,827 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>>> 2013-07-12 11:04:19,827 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>> 2013-07-12 11:04:19,829 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>> file
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>>> 2013-07-12 11:04:19,829 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 797
>>>>> 2013-07-12 11:04:26,002 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>> amount 104857600
>>>>> 2013-07-12 11:04:26,003 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>> available disk space. Entering safe mode.
>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>>>>> STATE* Safe mode is ON.
>>>>> Resources are low on NN. Please add or free up more resources then
>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode before
>>>>> adding resources, the NN will immediately return to safe mode..
>>>>> 2013-07-12 11:04:31,004 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>> amount 104857600
>>>>> 2013-07-12 11:04:31,005 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>> available disk space. Already in safe mode.
>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free up more
>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>>> 2013-07-12 11:04:36,005 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>> amount 104857600
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> N Venkata Rami Reddy
>>>> Hadoop Admin
>>>> Cloudwick Technologies
>>>>
>>>
>>>
>>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

I don't have it in my hdfs-site.xml, in which case probably the default
value is taken..


On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <az...@gmail.com> wrote:

> please check dfs.datanode.du.reserved in the hdfs-site.xml
> On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com> wrote:
>
>> Hi Krishna,
>>
>>    Can you please send screenshots of namenode web UI.
>>
>> Thanks Aditya.
>>
>>
>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> I have had enough space on the disk that is used, like around 30 Gigs
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
>>> venkatarami.netla@cloudwick.com> wrote:
>>>
>>>> Hi,
>>>> pls see the available space for NN storage directory.
>>>>
>>>> Thanks & Regards
>>>>
>>>> Venkat
>>>>
>>>>
>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>>>> write2kishore@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>>  I am doing no activity on my single node cluster which is using
>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>>>> after a while. I was looking at the name node log and see many of these
>>>>> kinds of entries.. Can anything be interpreted from these?
>>>>>
>>>>> 2013-07-12 09:06:11,256 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 561
>>>>> 2013-07-12 09:07:11,290 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 09:07:11,290 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 09:07:11,290 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>>> 2013-07-12 09:07:11,291 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>>> 2013-07-12 09:07:11,292 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>>> 2013-07-12 09:07:11,293 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>> file
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>>> 2013-07-12 09:07:11,294 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 563
>>>>> 2013-07-12 09:08:11,397 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 09:08:11,398 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 09:08:11,398 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>>> 2013-07-12 09:08:11,399 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>>> 2013-07-12 09:08:11,400 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>> 2013-07-12 09:08:11,402 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>> file
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>>> 2013-07-12 09:08:11,402 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 565
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>>> 2013-07-12 09:09:11,440 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>>> 2013-07-12 09:09:11,441 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>>
>>>>>
>>>>> And after sometime it said:
>>>>>
>>>>> 2013-07-12 11:03:19,799 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 795
>>>>> 2013-07-12 11:04:19,826 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>>> 9.70.137.114
>>>>> 2013-07-12 11:04:19,826 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>>> 2013-07-12 11:04:19,827 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>>> 2013-07-12 11:04:19,827 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>>> 2013-07-12 11:04:19,827 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>>> 2013-07-12 11:04:19,829 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>>> file
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>>> 2013-07-12 11:04:19,829 INFO
>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>>> 797
>>>>> 2013-07-12 11:04:26,002 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>> amount 104857600
>>>>> 2013-07-12 11:04:26,003 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>> available disk space. Entering safe mode.
>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange:
>>>>> STATE* Safe mode is ON.
>>>>> Resources are low on NN. Please add or free up more resources then
>>>>> turn off safe mode manually.  NOTE:  If you turn off safe mode before
>>>>> adding resources, the NN will immediately return to safe mode..
>>>>> 2013-07-12 11:04:31,004 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>> amount 104857600
>>>>> 2013-07-12 11:04:31,005 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>>> available disk space. Already in safe mode.
>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange:
>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free up more
>>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>>> 2013-07-12 11:04:36,005 WARN
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>>> available on volume 'null' is 0, which is below the configured reserved
>>>>> amount 104857600
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> N Venkata Rami Reddy
>>>> Hadoop Admin
>>>> Cloudwick Technologies
>>>>
>>>
>>>
>>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Azuryy Yu <az...@gmail.com>.

please check dfs.datanode.du.reserved in the hdfs-site.xml
On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com> wrote:

> Hi Krishna,
>
>    Can you please send screenshots of namenode web UI.
>
> Thanks Aditya.
>
>
> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> I have had enough space on the disk that is used, like around 30 Gigs
>>
>> Thanks,
>> Kishore
>>
>>
>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
>> venkatarami.netla@cloudwick.com> wrote:
>>
>>> Hi,
>>> pls see the available space for NN storage directory.
>>>
>>> Thanks & Regards
>>>
>>> Venkat
>>>
>>>
>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>>  I am doing no activity on my single node cluster which is using
>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>>> after a while. I was looking at the name node log and see many of these
>>>> kinds of entries.. Can anything be interpreted from these?
>>>>
>>>> 2013-07-12 09:06:11,256 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 561
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>> 2013-07-12 09:07:11,291 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>> 2013-07-12 09:07:11,292 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>> 2013-07-12 09:07:11,293 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>> 2013-07-12 09:07:11,294 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 563
>>>> 2013-07-12 09:08:11,397 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>> 2013-07-12 09:08:11,399 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>> 2013-07-12 09:08:11,400 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>> 2013-07-12 09:09:11,441 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>
>>>>
>>>> And after sometime it said:
>>>>
>>>> 2013-07-12 11:03:19,799 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 795
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 797
>>>> 2013-07-12 11:04:26,002 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:26,003 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Entering safe mode.
>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ON.
>>>> Resources are low on NN. Please add or free up more resources then turn
>>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>>> resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:31,004 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:31,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Already in safe mode.
>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ONResources are low on NN. Please add or free up more
>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:36,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>>
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>
>>>
>>>
>>> --
>>> N Venkata Rami Reddy
>>> Hadoop Admin
>>> Cloudwick Technologies
>>>
>>
>>
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Azuryy Yu <az...@gmail.com>.

please check dfs.datanode.du.reserved in the hdfs-site.xml
On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com> wrote:

> Hi Krishna,
>
>    Can you please send screenshots of namenode web UI.
>
> Thanks Aditya.
>
>
> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> I have had enough space on the disk that is used, like around 30 Gigs
>>
>> Thanks,
>> Kishore
>>
>>
>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
>> venkatarami.netla@cloudwick.com> wrote:
>>
>>> Hi,
>>> pls see the available space for NN storage directory.
>>>
>>> Thanks & Regards
>>>
>>> Venkat
>>>
>>>
>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>>  I am doing no activity on my single node cluster which is using
>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>>> after a while. I was looking at the name node log and see many of these
>>>> kinds of entries.. Can anything be interpreted from these?
>>>>
>>>> 2013-07-12 09:06:11,256 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 561
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>> 2013-07-12 09:07:11,291 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>> 2013-07-12 09:07:11,292 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>> 2013-07-12 09:07:11,293 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>> 2013-07-12 09:07:11,294 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 563
>>>> 2013-07-12 09:08:11,397 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>> 2013-07-12 09:08:11,399 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>> 2013-07-12 09:08:11,400 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>> 2013-07-12 09:09:11,441 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>
>>>>
>>>> And after sometime it said:
>>>>
>>>> 2013-07-12 11:03:19,799 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 795
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 797
>>>> 2013-07-12 11:04:26,002 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:26,003 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Entering safe mode.
>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ON.
>>>> Resources are low on NN. Please add or free up more resources then turn
>>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>>> resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:31,004 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:31,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Already in safe mode.
>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ONResources are low on NN. Please add or free up more
>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:36,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>>
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>
>>>
>>>
>>> --
>>> N Venkata Rami Reddy
>>> Hadoop Admin
>>> Cloudwick Technologies
>>>
>>
>>
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Azuryy Yu <az...@gmail.com>.

please check dfs.datanode.du.reserved in the hdfs-site.xml
On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com> wrote:

> Hi Krishna,
>
>    Can you please send screenshots of namenode web UI.
>
> Thanks Aditya.
>
>
> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> I have had enough space on the disk that is used, like around 30 Gigs
>>
>> Thanks,
>> Kishore
>>
>>
>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
>> venkatarami.netla@cloudwick.com> wrote:
>>
>>> Hi,
>>> pls see the available space for NN storage directory.
>>>
>>> Thanks & Regards
>>>
>>> Venkat
>>>
>>>
>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>>  I am doing no activity on my single node cluster which is using
>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>>> after a while. I was looking at the name node log and see many of these
>>>> kinds of entries.. Can anything be interpreted from these?
>>>>
>>>> 2013-07-12 09:06:11,256 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 561
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>> 2013-07-12 09:07:11,291 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>> 2013-07-12 09:07:11,292 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>> 2013-07-12 09:07:11,293 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>> 2013-07-12 09:07:11,294 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 563
>>>> 2013-07-12 09:08:11,397 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>> 2013-07-12 09:08:11,399 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>> 2013-07-12 09:08:11,400 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>> 2013-07-12 09:09:11,441 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>
>>>>
>>>> And after sometime it said:
>>>>
>>>> 2013-07-12 11:03:19,799 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 795
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 797
>>>> 2013-07-12 11:04:26,002 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:26,003 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Entering safe mode.
>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ON.
>>>> Resources are low on NN. Please add or free up more resources then turn
>>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>>> resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:31,004 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:31,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Already in safe mode.
>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ONResources are low on NN. Please add or free up more
>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:36,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>>
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>
>>>
>>>
>>> --
>>> N Venkata Rami Reddy
>>> Hadoop Admin
>>> Cloudwick Technologies
>>>
>>
>>
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Azuryy Yu <az...@gmail.com>.

please check dfs.datanode.du.reserved in the hdfs-site.xml
On Jul 15, 2013 4:30 PM, "Aditya exalter" <ad...@gmail.com> wrote:

> Hi Krishna,
>
>    Can you please send screenshots of namenode web UI.
>
> Thanks Aditya.
>
>
> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> I have had enough space on the disk that is used, like around 30 Gigs
>>
>> Thanks,
>> Kishore
>>
>>
>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
>> venkatarami.netla@cloudwick.com> wrote:
>>
>>> Hi,
>>> pls see the available space for NN storage directory.
>>>
>>> Thanks & Regards
>>>
>>> Venkat
>>>
>>>
>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>>  I am doing no activity on my single node cluster which is using
>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>>> after a while. I was looking at the name node log and see many of these
>>>> kinds of entries.. Can anything be interpreted from these?
>>>>
>>>> 2013-07-12 09:06:11,256 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 561
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>> 2013-07-12 09:07:11,291 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>> 2013-07-12 09:07:11,292 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>> 2013-07-12 09:07:11,293 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>> 2013-07-12 09:07:11,294 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 563
>>>> 2013-07-12 09:08:11,397 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>> 2013-07-12 09:08:11,399 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>> 2013-07-12 09:08:11,400 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>> 2013-07-12 09:09:11,441 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>
>>>>
>>>> And after sometime it said:
>>>>
>>>> 2013-07-12 11:03:19,799 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 795
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 797
>>>> 2013-07-12 11:04:26,002 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:26,003 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Entering safe mode.
>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ON.
>>>> Resources are low on NN. Please add or free up more resources then turn
>>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>>> resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:31,004 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:31,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Already in safe mode.
>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ONResources are low on NN. Please add or free up more
>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:36,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>>
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>
>>>
>>>
>>> --
>>> N Venkata Rami Reddy
>>> Hadoop Admin
>>> Cloudwick Technologies
>>>
>>
>>
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Aditya exalter <ad...@gmail.com>.

Hi Krishna,

   Can you please send screenshots of namenode web UI.

Thanks Aditya.


On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> I have had enough space on the disk that is used, like around 30 Gigs
>
> Thanks,
> Kishore
>
>
> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
> venkatarami.netla@cloudwick.com> wrote:
>
>> Hi,
>> pls see the available space for NN storage directory.
>>
>> Thanks & Regards
>>
>> Venkat
>>
>>
>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>>  I am doing no activity on my single node cluster which is using
>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>> after a while. I was looking at the name node log and see many of these
>>> kinds of entries.. Can anything be interpreted from these?
>>>
>>> 2013-07-12 09:06:11,256 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 561
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>> 2013-07-12 09:07:11,291 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>> 2013-07-12 09:07:11,292 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>> 2013-07-12 09:07:11,293 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>> 2013-07-12 09:07:11,294 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 563
>>> 2013-07-12 09:08:11,397 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>> 2013-07-12 09:08:11,399 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>> 2013-07-12 09:08:11,400 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>> 2013-07-12 09:09:11,441 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>
>>>
>>> And after sometime it said:
>>>
>>> 2013-07-12 11:03:19,799 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 795
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 797
>>> 2013-07-12 11:04:26,002 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:26,003 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Entering safe mode.
>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ON.
>>> Resources are low on NN. Please add or free up more resources then turn
>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>> resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:31,004 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:31,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Already in safe mode.
>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ONResources are low on NN. Please add or free up more
>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>> mode before adding resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:36,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>>
>> --
>> N Venkata Rami Reddy
>> Hadoop Admin
>> Cloudwick Technologies
>>
>
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Aditya exalter <ad...@gmail.com>.

Hi Krishna,

   Can you please send screenshots of namenode web UI.

Thanks Aditya.


On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> I have had enough space on the disk that is used, like around 30 Gigs
>
> Thanks,
> Kishore
>
>
> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
> venkatarami.netla@cloudwick.com> wrote:
>
>> Hi,
>> pls see the available space for NN storage directory.
>>
>> Thanks & Regards
>>
>> Venkat
>>
>>
>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>>  I am doing no activity on my single node cluster which is using
>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>> after a while. I was looking at the name node log and see many of these
>>> kinds of entries.. Can anything be interpreted from these?
>>>
>>> 2013-07-12 09:06:11,256 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 561
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>> 2013-07-12 09:07:11,291 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>> 2013-07-12 09:07:11,292 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>> 2013-07-12 09:07:11,293 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>> 2013-07-12 09:07:11,294 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 563
>>> 2013-07-12 09:08:11,397 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>> 2013-07-12 09:08:11,399 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>> 2013-07-12 09:08:11,400 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>> 2013-07-12 09:09:11,441 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>
>>>
>>> And after sometime it said:
>>>
>>> 2013-07-12 11:03:19,799 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 795
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 797
>>> 2013-07-12 11:04:26,002 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:26,003 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Entering safe mode.
>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ON.
>>> Resources are low on NN. Please add or free up more resources then turn
>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>> resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:31,004 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:31,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Already in safe mode.
>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ONResources are low on NN. Please add or free up more
>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>> mode before adding resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:36,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>>
>> --
>> N Venkata Rami Reddy
>> Hadoop Admin
>> Cloudwick Technologies
>>
>
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Aditya exalter <ad...@gmail.com>.

Hi Krishna,

   Can you please send screenshots of namenode web UI.

Thanks Aditya.


On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> I have had enough space on the disk that is used, like around 30 Gigs
>
> Thanks,
> Kishore
>
>
> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
> venkatarami.netla@cloudwick.com> wrote:
>
>> Hi,
>> pls see the available space for NN storage directory.
>>
>> Thanks & Regards
>>
>> Venkat
>>
>>
>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>>  I am doing no activity on my single node cluster which is using
>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>> after a while. I was looking at the name node log and see many of these
>>> kinds of entries.. Can anything be interpreted from these?
>>>
>>> 2013-07-12 09:06:11,256 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 561
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>> 2013-07-12 09:07:11,291 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>> 2013-07-12 09:07:11,292 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>> 2013-07-12 09:07:11,293 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>> 2013-07-12 09:07:11,294 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 563
>>> 2013-07-12 09:08:11,397 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>> 2013-07-12 09:08:11,399 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>> 2013-07-12 09:08:11,400 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>> 2013-07-12 09:09:11,441 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>
>>>
>>> And after sometime it said:
>>>
>>> 2013-07-12 11:03:19,799 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 795
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 797
>>> 2013-07-12 11:04:26,002 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:26,003 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Entering safe mode.
>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ON.
>>> Resources are low on NN. Please add or free up more resources then turn
>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>> resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:31,004 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:31,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Already in safe mode.
>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ONResources are low on NN. Please add or free up more
>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>> mode before adding resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:36,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>>
>> --
>> N Venkata Rami Reddy
>> Hadoop Admin
>> Cloudwick Technologies
>>
>
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi,

  I have restarted my cluster after removing the data directory and
formatting the namenode. So, is this screenshot still useful for you or do
you want it only after I reproduce the issue?

Thanks,
Kishore


On Mon, Jul 15, 2013 at 1:59 PM, Venkatarami Netla <
venkatarami.netla@cloudwick.com> wrote:

> Hi,
> Pls send Web UI namemnode page and what is ur total hard disk size
>
>
> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> I have had enough space on the disk that is used, like around 30 Gigs
>>
>> Thanks,
>> Kishore
>>
>>
>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
>> venkatarami.netla@cloudwick.com> wrote:
>>
>>> Hi,
>>> pls see the available space for NN storage directory.
>>>
>>> Thanks & Regards
>>>
>>> Venkat
>>>
>>>
>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>>  I am doing no activity on my single node cluster which is using
>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>>> after a while. I was looking at the name node log and see many of these
>>>> kinds of entries.. Can anything be interpreted from these?
>>>>
>>>> 2013-07-12 09:06:11,256 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 561
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>> 2013-07-12 09:07:11,291 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>> 2013-07-12 09:07:11,292 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>> 2013-07-12 09:07:11,293 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>> 2013-07-12 09:07:11,294 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 563
>>>> 2013-07-12 09:08:11,397 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>> 2013-07-12 09:08:11,399 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>> 2013-07-12 09:08:11,400 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>> 2013-07-12 09:09:11,441 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>
>>>>
>>>> And after sometime it said:
>>>>
>>>> 2013-07-12 11:03:19,799 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 795
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 797
>>>> 2013-07-12 11:04:26,002 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:26,003 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Entering safe mode.
>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ON.
>>>> Resources are low on NN. Please add or free up more resources then turn
>>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>>> resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:31,004 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:31,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Already in safe mode.
>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ONResources are low on NN. Please add or free up more
>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:36,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>>
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>
>>>
>>>
>>> --
>>> N Venkata Rami Reddy
>>> Hadoop Admin
>>> Cloudwick Technologies
>>>
>>
>>
>
>
> --
> N Venkata Rami Reddy
> Hadoop Admin
> Cloudwick Technologies
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi,

  I have restarted my cluster after removing the data directory and
formatting the namenode. So, is this screenshot still useful for you or do
you want it only after I reproduce the issue?

Thanks,
Kishore


On Mon, Jul 15, 2013 at 1:59 PM, Venkatarami Netla <
venkatarami.netla@cloudwick.com> wrote:

> Hi,
> Pls send Web UI namemnode page and what is ur total hard disk size
>
>
> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> I have had enough space on the disk that is used, like around 30 Gigs
>>
>> Thanks,
>> Kishore
>>
>>
>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
>> venkatarami.netla@cloudwick.com> wrote:
>>
>>> Hi,
>>> pls see the available space for NN storage directory.
>>>
>>> Thanks & Regards
>>>
>>> Venkat
>>>
>>>
>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>>  I am doing no activity on my single node cluster which is using
>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>>> after a while. I was looking at the name node log and see many of these
>>>> kinds of entries.. Can anything be interpreted from these?
>>>>
>>>> 2013-07-12 09:06:11,256 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 561
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>> 2013-07-12 09:07:11,291 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>> 2013-07-12 09:07:11,292 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>> 2013-07-12 09:07:11,293 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>> 2013-07-12 09:07:11,294 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 563
>>>> 2013-07-12 09:08:11,397 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>> 2013-07-12 09:08:11,399 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>> 2013-07-12 09:08:11,400 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>> 2013-07-12 09:09:11,441 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>
>>>>
>>>> And after sometime it said:
>>>>
>>>> 2013-07-12 11:03:19,799 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 795
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 797
>>>> 2013-07-12 11:04:26,002 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:26,003 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Entering safe mode.
>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ON.
>>>> Resources are low on NN. Please add or free up more resources then turn
>>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>>> resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:31,004 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:31,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Already in safe mode.
>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ONResources are low on NN. Please add or free up more
>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:36,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>>
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>
>>>
>>>
>>> --
>>> N Venkata Rami Reddy
>>> Hadoop Admin
>>> Cloudwick Technologies
>>>
>>
>>
>
>
> --
> N Venkata Rami Reddy
> Hadoop Admin
> Cloudwick Technologies
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi,

  I have restarted my cluster after removing the data directory and
formatting the namenode. So, is this screenshot still useful for you or do
you want it only after I reproduce the issue?

Thanks,
Kishore


On Mon, Jul 15, 2013 at 1:59 PM, Venkatarami Netla <
venkatarami.netla@cloudwick.com> wrote:

> Hi,
> Pls send Web UI namemnode page and what is ur total hard disk size
>
>
> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> I have had enough space on the disk that is used, like around 30 Gigs
>>
>> Thanks,
>> Kishore
>>
>>
>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
>> venkatarami.netla@cloudwick.com> wrote:
>>
>>> Hi,
>>> pls see the available space for NN storage directory.
>>>
>>> Thanks & Regards
>>>
>>> Venkat
>>>
>>>
>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>>  I am doing no activity on my single node cluster which is using
>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>>> after a while. I was looking at the name node log and see many of these
>>>> kinds of entries.. Can anything be interpreted from these?
>>>>
>>>> 2013-07-12 09:06:11,256 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 561
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>> 2013-07-12 09:07:11,291 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>> 2013-07-12 09:07:11,292 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>> 2013-07-12 09:07:11,293 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>> 2013-07-12 09:07:11,294 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 563
>>>> 2013-07-12 09:08:11,397 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>> 2013-07-12 09:08:11,399 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>> 2013-07-12 09:08:11,400 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>> 2013-07-12 09:09:11,441 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>
>>>>
>>>> And after sometime it said:
>>>>
>>>> 2013-07-12 11:03:19,799 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 795
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 797
>>>> 2013-07-12 11:04:26,002 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:26,003 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Entering safe mode.
>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ON.
>>>> Resources are low on NN. Please add or free up more resources then turn
>>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>>> resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:31,004 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:31,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Already in safe mode.
>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ONResources are low on NN. Please add or free up more
>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:36,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>>
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>
>>>
>>>
>>> --
>>> N Venkata Rami Reddy
>>> Hadoop Admin
>>> Cloudwick Technologies
>>>
>>
>>
>
>
> --
> N Venkata Rami Reddy
> Hadoop Admin
> Cloudwick Technologies
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi,

  I have restarted my cluster after removing the data directory and
formatting the namenode. So, is this screenshot still useful for you or do
you want it only after I reproduce the issue?

Thanks,
Kishore


On Mon, Jul 15, 2013 at 1:59 PM, Venkatarami Netla <
venkatarami.netla@cloudwick.com> wrote:

> Hi,
> Pls send Web UI namemnode page and what is ur total hard disk size
>
>
> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> I have had enough space on the disk that is used, like around 30 Gigs
>>
>> Thanks,
>> Kishore
>>
>>
>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
>> venkatarami.netla@cloudwick.com> wrote:
>>
>>> Hi,
>>> pls see the available space for NN storage directory.
>>>
>>> Thanks & Regards
>>>
>>> Venkat
>>>
>>>
>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>>  I am doing no activity on my single node cluster which is using
>>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>>> after a while. I was looking at the name node log and see many of these
>>>> kinds of entries.. Can anything be interpreted from these?
>>>>
>>>> 2013-07-12 09:06:11,256 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 561
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:07:11,290 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>>> 2013-07-12 09:07:11,291 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>>> 2013-07-12 09:07:11,292 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>>> 2013-07-12 09:07:11,293 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>>> 2013-07-12 09:07:11,294 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 563
>>>> 2013-07-12 09:08:11,397 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:08:11,398 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>>> 2013-07-12 09:08:11,399 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>>> 2013-07-12 09:08:11,400 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>>> 2013-07-12 09:08:11,402 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>>> 2013-07-12 09:09:11,440 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>>> 2013-07-12 09:09:11,441 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>>
>>>>
>>>> And after sometime it said:
>>>>
>>>> 2013-07-12 11:03:19,799 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 795
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>>> 9.70.137.114
>>>> 2013-07-12 11:04:19,826 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,827 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>>> file
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>>> 2013-07-12 11:04:19,829 INFO
>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>>> 797
>>>> 2013-07-12 11:04:26,002 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:26,003 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Entering safe mode.
>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ON.
>>>> Resources are low on NN. Please add or free up more resources then turn
>>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>>> resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:31,004 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>> 2013-07-12 11:04:31,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>>> available disk space. Already in safe mode.
>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>>> Safe mode is ONResources are low on NN. Please add or free up more
>>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>>> mode before adding resources, the NN will immediately return to safe mode..
>>>> 2013-07-12 11:04:36,005 WARN
>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>>> available on volume 'null' is 0, which is below the configured reserved
>>>> amount 104857600
>>>>
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>
>>>
>>>
>>> --
>>> N Venkata Rami Reddy
>>> Hadoop Admin
>>> Cloudwick Technologies
>>>
>>
>>
>
>
> --
> N Venkata Rami Reddy
> Hadoop Admin
> Cloudwick Technologies
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Venkatarami Netla <ve...@cloudwick.com>.

Hi,
Pls send Web UI namemnode page and what is ur total hard disk size


On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> I have had enough space on the disk that is used, like around 30 Gigs
>
> Thanks,
> Kishore
>
>
> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
> venkatarami.netla@cloudwick.com> wrote:
>
>> Hi,
>> pls see the available space for NN storage directory.
>>
>> Thanks & Regards
>>
>> Venkat
>>
>>
>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>>  I am doing no activity on my single node cluster which is using
>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>> after a while. I was looking at the name node log and see many of these
>>> kinds of entries.. Can anything be interpreted from these?
>>>
>>> 2013-07-12 09:06:11,256 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 561
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>> 2013-07-12 09:07:11,291 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>> 2013-07-12 09:07:11,292 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>> 2013-07-12 09:07:11,293 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>> 2013-07-12 09:07:11,294 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 563
>>> 2013-07-12 09:08:11,397 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>> 2013-07-12 09:08:11,399 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>> 2013-07-12 09:08:11,400 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>> 2013-07-12 09:09:11,441 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>
>>>
>>> And after sometime it said:
>>>
>>> 2013-07-12 11:03:19,799 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 795
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 797
>>> 2013-07-12 11:04:26,002 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:26,003 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Entering safe mode.
>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ON.
>>> Resources are low on NN. Please add or free up more resources then turn
>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>> resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:31,004 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:31,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Already in safe mode.
>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ONResources are low on NN. Please add or free up more
>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>> mode before adding resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:36,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>>
>> --
>> N Venkata Rami Reddy
>> Hadoop Admin
>> Cloudwick Technologies
>>
>
>


-- 
N Venkata Rami Reddy
Hadoop Admin
Cloudwick Technologies

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Venkatarami Netla <ve...@cloudwick.com>.

Hi,
Pls send Web UI namemnode page and what is ur total hard disk size


On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> I have had enough space on the disk that is used, like around 30 Gigs
>
> Thanks,
> Kishore
>
>
> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
> venkatarami.netla@cloudwick.com> wrote:
>
>> Hi,
>> pls see the available space for NN storage directory.
>>
>> Thanks & Regards
>>
>> Venkat
>>
>>
>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>>  I am doing no activity on my single node cluster which is using
>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>> after a while. I was looking at the name node log and see many of these
>>> kinds of entries.. Can anything be interpreted from these?
>>>
>>> 2013-07-12 09:06:11,256 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 561
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>> 2013-07-12 09:07:11,291 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>> 2013-07-12 09:07:11,292 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>> 2013-07-12 09:07:11,293 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>> 2013-07-12 09:07:11,294 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 563
>>> 2013-07-12 09:08:11,397 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>> 2013-07-12 09:08:11,399 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>> 2013-07-12 09:08:11,400 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>> 2013-07-12 09:09:11,441 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>
>>>
>>> And after sometime it said:
>>>
>>> 2013-07-12 11:03:19,799 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 795
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 797
>>> 2013-07-12 11:04:26,002 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:26,003 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Entering safe mode.
>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ON.
>>> Resources are low on NN. Please add or free up more resources then turn
>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>> resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:31,004 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:31,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Already in safe mode.
>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ONResources are low on NN. Please add or free up more
>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>> mode before adding resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:36,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>>
>> --
>> N Venkata Rami Reddy
>> Hadoop Admin
>> Cloudwick Technologies
>>
>
>


-- 
N Venkata Rami Reddy
Hadoop Admin
Cloudwick Technologies

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Venkatarami Netla <ve...@cloudwick.com>.

Hi,
Pls send Web UI namemnode page and what is ur total hard disk size


On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> I have had enough space on the disk that is used, like around 30 Gigs
>
> Thanks,
> Kishore
>
>
> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
> venkatarami.netla@cloudwick.com> wrote:
>
>> Hi,
>> pls see the available space for NN storage directory.
>>
>> Thanks & Regards
>>
>> Venkat
>>
>>
>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>>  I am doing no activity on my single node cluster which is using
>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>> after a while. I was looking at the name node log and see many of these
>>> kinds of entries.. Can anything be interpreted from these?
>>>
>>> 2013-07-12 09:06:11,256 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 561
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>> 2013-07-12 09:07:11,291 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>> 2013-07-12 09:07:11,292 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>> 2013-07-12 09:07:11,293 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>> 2013-07-12 09:07:11,294 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 563
>>> 2013-07-12 09:08:11,397 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>> 2013-07-12 09:08:11,399 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>> 2013-07-12 09:08:11,400 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>> 2013-07-12 09:09:11,441 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>
>>>
>>> And after sometime it said:
>>>
>>> 2013-07-12 11:03:19,799 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 795
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 797
>>> 2013-07-12 11:04:26,002 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:26,003 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Entering safe mode.
>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ON.
>>> Resources are low on NN. Please add or free up more resources then turn
>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>> resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:31,004 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:31,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Already in safe mode.
>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ONResources are low on NN. Please add or free up more
>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>> mode before adding resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:36,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>>
>> --
>> N Venkata Rami Reddy
>> Hadoop Admin
>> Cloudwick Technologies
>>
>
>


-- 
N Venkata Rami Reddy
Hadoop Admin
Cloudwick Technologies

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Aditya exalter <ad...@gmail.com>.

Hi Krishna,

   Can you please send screenshots of namenode web UI.

Thanks Aditya.


On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> I have had enough space on the disk that is used, like around 30 Gigs
>
> Thanks,
> Kishore
>
>
> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
> venkatarami.netla@cloudwick.com> wrote:
>
>> Hi,
>> pls see the available space for NN storage directory.
>>
>> Thanks & Regards
>>
>> Venkat
>>
>>
>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>>  I am doing no activity on my single node cluster which is using
>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>> after a while. I was looking at the name node log and see many of these
>>> kinds of entries.. Can anything be interpreted from these?
>>>
>>> 2013-07-12 09:06:11,256 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 561
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>> 2013-07-12 09:07:11,291 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>> 2013-07-12 09:07:11,292 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>> 2013-07-12 09:07:11,293 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>> 2013-07-12 09:07:11,294 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 563
>>> 2013-07-12 09:08:11,397 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>> 2013-07-12 09:08:11,399 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>> 2013-07-12 09:08:11,400 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>> 2013-07-12 09:09:11,441 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>
>>>
>>> And after sometime it said:
>>>
>>> 2013-07-12 11:03:19,799 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 795
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 797
>>> 2013-07-12 11:04:26,002 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:26,003 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Entering safe mode.
>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ON.
>>> Resources are low on NN. Please add or free up more resources then turn
>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>> resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:31,004 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:31,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Already in safe mode.
>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ONResources are low on NN. Please add or free up more
>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>> mode before adding resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:36,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>>
>> --
>> N Venkata Rami Reddy
>> Hadoop Admin
>> Cloudwick Technologies
>>
>
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Venkatarami Netla <ve...@cloudwick.com>.

Hi,
Pls send Web UI namemnode page and what is ur total hard disk size


On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> I have had enough space on the disk that is used, like around 30 Gigs
>
> Thanks,
> Kishore
>
>
> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
> venkatarami.netla@cloudwick.com> wrote:
>
>> Hi,
>> pls see the available space for NN storage directory.
>>
>> Thanks & Regards
>>
>> Venkat
>>
>>
>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>>  I am doing no activity on my single node cluster which is using
>>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>>> after a while. I was looking at the name node log and see many of these
>>> kinds of entries.. Can anything be interpreted from these?
>>>
>>> 2013-07-12 09:06:11,256 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 561
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:07:11,290 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>>> 2013-07-12 09:07:11,291 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 14
>>> 2013-07-12 09:07:11,292 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 15
>>> 2013-07-12 09:07:11,293 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>>> 2013-07-12 09:07:11,294 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 563
>>> 2013-07-12 09:08:11,397 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:08:11,398 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>>> 2013-07-12 09:08:11,399 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 11
>>> 2013-07-12 09:08:11,400 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>>> 2013-07-12 09:08:11,402 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>>> 2013-07-12 09:09:11,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 13
>>> 2013-07-12 09:09:11,441 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>>
>>>
>>> And after sometime it said:
>>>
>>> 2013-07-12 11:03:19,799 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 795
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>>> 9.70.137.114
>>> 2013-07-12 11:04:19,826 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 2 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,827 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>>> 0 Number of syncs: 3 SyncTimes(ms): 12
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>>> file
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>>> 2013-07-12 11:04:19,829 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>>> 797
>>> 2013-07-12 11:04:26,002 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:26,003 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Entering safe mode.
>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ON.
>>> Resources are low on NN. Please add or free up more resources then turn
>>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>>> resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:31,004 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>> 2013-07-12 11:04:31,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>>> available disk space. Already in safe mode.
>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>>> Safe mode is ONResources are low on NN. Please add or free up more
>>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>>> mode before adding resources, the NN will immediately return to safe mode..
>>> 2013-07-12 11:04:36,005 WARN
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>>> available on volume 'null' is 0, which is below the configured reserved
>>> amount 104857600
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>>
>> --
>> N Venkata Rami Reddy
>> Hadoop Admin
>> Cloudwick Technologies
>>
>
>


-- 
N Venkata Rami Reddy
Hadoop Admin
Cloudwick Technologies

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

I have had enough space on the disk that is used, like around 30 Gigs

Thanks,
Kishore


On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
venkatarami.netla@cloudwick.com> wrote:

> Hi,
> pls see the available space for NN storage directory.
>
> Thanks & Regards
>
> Venkat
>
>
> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>
>>  I am doing no activity on my single node cluster which is using
>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>> after a while. I was looking at the name node log and see many of these
>> kinds of entries.. Can anything be interpreted from these?
>>
>> 2013-07-12 09:06:11,256 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 561
>> 2013-07-12 09:07:11,290 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 09:07:11,290 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 09:07:11,290 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>> 2013-07-12 09:07:11,291 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 14
>> 2013-07-12 09:07:11,292 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 15
>> 2013-07-12 09:07:11,293 INFO
>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>> 2013-07-12 09:07:11,294 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 563
>> 2013-07-12 09:08:11,397 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 09:08:11,398 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 09:08:11,398 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>> 2013-07-12 09:08:11,399 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 11
>> 2013-07-12 09:08:11,400 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> 2013-07-12 09:08:11,402 INFO
>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>> 2013-07-12 09:08:11,402 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 565
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 13
>> 2013-07-12 09:09:11,441 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>
>>
>> And after sometime it said:
>>
>> 2013-07-12 11:03:19,799 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 795
>> 2013-07-12 11:04:19,826 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 11:04:19,826 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 11:04:19,827 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>> 2013-07-12 11:04:19,827 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 12
>> 2013-07-12 11:04:19,827 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> 2013-07-12 11:04:19,829 INFO
>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>> 2013-07-12 11:04:19,829 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 797
>> 2013-07-12 11:04:26,002 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>> 2013-07-12 11:04:26,003 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>> available disk space. Entering safe mode.
>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>> Safe mode is ON.
>> Resources are low on NN. Please add or free up more resources then turn
>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>> resources, the NN will immediately return to safe mode..
>> 2013-07-12 11:04:31,004 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>> 2013-07-12 11:04:31,005 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>> available disk space. Already in safe mode.
>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>> Safe mode is ONResources are low on NN. Please add or free up more
>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>> mode before adding resources, the NN will immediately return to safe mode..
>> 2013-07-12 11:04:36,005 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>>
>>
>> Thanks,
>> Kishore
>>
>
>
>
> --
> N Venkata Rami Reddy
> Hadoop Admin
> Cloudwick Technologies
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

I have had enough space on the disk that is used, like around 30 Gigs

Thanks,
Kishore


On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
venkatarami.netla@cloudwick.com> wrote:

> Hi,
> pls see the available space for NN storage directory.
>
> Thanks & Regards
>
> Venkat
>
>
> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>
>>  I am doing no activity on my single node cluster which is using
>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>> after a while. I was looking at the name node log and see many of these
>> kinds of entries.. Can anything be interpreted from these?
>>
>> 2013-07-12 09:06:11,256 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 561
>> 2013-07-12 09:07:11,290 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 09:07:11,290 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 09:07:11,290 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>> 2013-07-12 09:07:11,291 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 14
>> 2013-07-12 09:07:11,292 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 15
>> 2013-07-12 09:07:11,293 INFO
>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>> 2013-07-12 09:07:11,294 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 563
>> 2013-07-12 09:08:11,397 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 09:08:11,398 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 09:08:11,398 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>> 2013-07-12 09:08:11,399 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 11
>> 2013-07-12 09:08:11,400 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> 2013-07-12 09:08:11,402 INFO
>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>> 2013-07-12 09:08:11,402 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 565
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 13
>> 2013-07-12 09:09:11,441 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>
>>
>> And after sometime it said:
>>
>> 2013-07-12 11:03:19,799 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 795
>> 2013-07-12 11:04:19,826 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 11:04:19,826 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 11:04:19,827 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>> 2013-07-12 11:04:19,827 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 12
>> 2013-07-12 11:04:19,827 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> 2013-07-12 11:04:19,829 INFO
>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>> 2013-07-12 11:04:19,829 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 797
>> 2013-07-12 11:04:26,002 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>> 2013-07-12 11:04:26,003 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>> available disk space. Entering safe mode.
>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>> Safe mode is ON.
>> Resources are low on NN. Please add or free up more resources then turn
>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>> resources, the NN will immediately return to safe mode..
>> 2013-07-12 11:04:31,004 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>> 2013-07-12 11:04:31,005 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>> available disk space. Already in safe mode.
>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>> Safe mode is ONResources are low on NN. Please add or free up more
>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>> mode before adding resources, the NN will immediately return to safe mode..
>> 2013-07-12 11:04:36,005 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>>
>>
>> Thanks,
>> Kishore
>>
>
>
>
> --
> N Venkata Rami Reddy
> Hadoop Admin
> Cloudwick Technologies
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

I have had enough space on the disk that is used, like around 30 Gigs

Thanks,
Kishore


On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
venkatarami.netla@cloudwick.com> wrote:

> Hi,
> pls see the available space for NN storage directory.
>
> Thanks & Regards
>
> Venkat
>
>
> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>
>>  I am doing no activity on my single node cluster which is using
>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>> after a while. I was looking at the name node log and see many of these
>> kinds of entries.. Can anything be interpreted from these?
>>
>> 2013-07-12 09:06:11,256 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 561
>> 2013-07-12 09:07:11,290 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 09:07:11,290 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 09:07:11,290 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>> 2013-07-12 09:07:11,291 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 14
>> 2013-07-12 09:07:11,292 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 15
>> 2013-07-12 09:07:11,293 INFO
>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>> 2013-07-12 09:07:11,294 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 563
>> 2013-07-12 09:08:11,397 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 09:08:11,398 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 09:08:11,398 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>> 2013-07-12 09:08:11,399 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 11
>> 2013-07-12 09:08:11,400 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> 2013-07-12 09:08:11,402 INFO
>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>> 2013-07-12 09:08:11,402 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 565
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 13
>> 2013-07-12 09:09:11,441 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>
>>
>> And after sometime it said:
>>
>> 2013-07-12 11:03:19,799 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 795
>> 2013-07-12 11:04:19,826 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 11:04:19,826 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 11:04:19,827 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>> 2013-07-12 11:04:19,827 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 12
>> 2013-07-12 11:04:19,827 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> 2013-07-12 11:04:19,829 INFO
>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>> 2013-07-12 11:04:19,829 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 797
>> 2013-07-12 11:04:26,002 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>> 2013-07-12 11:04:26,003 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>> available disk space. Entering safe mode.
>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>> Safe mode is ON.
>> Resources are low on NN. Please add or free up more resources then turn
>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>> resources, the NN will immediately return to safe mode..
>> 2013-07-12 11:04:31,004 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>> 2013-07-12 11:04:31,005 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>> available disk space. Already in safe mode.
>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>> Safe mode is ONResources are low on NN. Please add or free up more
>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>> mode before adding resources, the NN will immediately return to safe mode..
>> 2013-07-12 11:04:36,005 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>>
>>
>> Thanks,
>> Kishore
>>
>
>
>
> --
> N Venkata Rami Reddy
> Hadoop Admin
> Cloudwick Technologies
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

I have had enough space on the disk that is used, like around 30 Gigs

Thanks,
Kishore


On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla <
venkatarami.netla@cloudwick.com> wrote:

> Hi,
> pls see the available space for NN storage directory.
>
> Thanks & Regards
>
> Venkat
>
>
> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>
>>  I am doing no activity on my single node cluster which is using
>> 2.1.0-beta, and still observed that it has gone to safe mode by itself
>> after a while. I was looking at the name node log and see many of these
>> kinds of entries.. Can anything be interpreted from these?
>>
>> 2013-07-12 09:06:11,256 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 561
>> 2013-07-12 09:07:11,290 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 09:07:11,290 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 09:07:11,290 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
>> 2013-07-12 09:07:11,291 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 14
>> 2013-07-12 09:07:11,292 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 15
>> 2013-07-12 09:07:11,293 INFO
>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
>> 2013-07-12 09:07:11,294 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 563
>> 2013-07-12 09:08:11,397 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 09:08:11,398 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 09:08:11,398 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
>> 2013-07-12 09:08:11,399 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 11
>> 2013-07-12 09:08:11,400 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> 2013-07-12 09:08:11,402 INFO
>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
>> 2013-07-12 09:08:11,402 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 565
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
>> 2013-07-12 09:09:11,440 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 13
>> 2013-07-12 09:09:11,441 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 13
>>
>>
>> And after sometime it said:
>>
>> 2013-07-12 11:03:19,799 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 795
>> 2013-07-12 11:04:19,826 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 9.70.137.114
>> 2013-07-12 11:04:19,826 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
>> 2013-07-12 11:04:19,827 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
>> 2013-07-12 11:04:19,827 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 2 SyncTimes(ms): 12
>> 2013-07-12 11:04:19,827 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
>> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
>> 0 Number of syncs: 3 SyncTimes(ms): 12
>> 2013-07-12 11:04:19,829 INFO
>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
>> file
>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
>> 2013-07-12 11:04:19,829 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
>> 797
>> 2013-07-12 11:04:26,002 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>> 2013-07-12 11:04:26,003 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>> available disk space. Entering safe mode.
>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>> Safe mode is ON.
>> Resources are low on NN. Please add or free up more resources then turn
>> off safe mode manually.  NOTE:  If you turn off safe mode before adding
>> resources, the NN will immediately return to safe mode..
>> 2013-07-12 11:04:31,004 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>> 2013-07-12 11:04:31,005 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
>> available disk space. Already in safe mode.
>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
>> Safe mode is ONResources are low on NN. Please add or free up more
>> resources then turn off safe mode manually.  NOTE:  If you turn off safe
>> mode before adding resources, the NN will immediately return to safe mode..
>> 2013-07-12 11:04:36,005 WARN
>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
>> available on volume 'null' is 0, which is below the configured reserved
>> amount 104857600
>>
>>
>> Thanks,
>> Kishore
>>
>
>
>
> --
> N Venkata Rami Reddy
> Hadoop Admin
> Cloudwick Technologies
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Venkatarami Netla <ve...@cloudwick.com>.

Hi,
pls see the available space for NN storage directory.

Thanks & Regards

Venkat


On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi,
>
>  I am doing no activity on my single node cluster which is using
> 2.1.0-beta, and still observed that it has gone to safe mode by itself
> after a while. I was looking at the name node log and see many of these
> kinds of entries.. Can anything be interpreted from these?
>
> 2013-07-12 09:06:11,256 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 561
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
> 2013-07-12 09:07:11,291 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 14
> 2013-07-12 09:07:11,292 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 15
> 2013-07-12 09:07:11,293 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> 2013-07-12 09:07:11,294 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 563
> 2013-07-12 09:08:11,397 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
> 2013-07-12 09:08:11,399 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 11
> 2013-07-12 09:08:11,400 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 13
> 2013-07-12 09:09:11,441 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 13
>
>
> And after sometime it said:
>
> 2013-07-12 11:03:19,799 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 795
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 12
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 797
> 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:26,003 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Entering safe mode.
> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ON.
> Resources are low on NN. Please add or free up more resources then turn
> off safe mode manually.  NOTE:  If you turn off safe mode before adding
> resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:31,004 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:31,005 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Already in safe mode.
> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ONResources are low on NN. Please add or free up more
> resources then turn off safe mode manually.  NOTE:  If you turn off safe
> mode before adding resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:36,005 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
>
> Thanks,
> Kishore
>



-- 
N Venkata Rami Reddy
Hadoop Admin
Cloudwick Technologies

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Azuryy Yu <az...@gmail.com>.

hi,
from the log:
NameNode low on available disk space. Entering safe mode.

this is the root cause.
 On Jul 15, 2013 2:45 PM, "Krishna Kishore Bonagiri" <
write2kishore@gmail.com> wrote:

> Hi,
>
>  I am doing no activity on my single node cluster which is using
> 2.1.0-beta, and still observed that it has gone to safe mode by itself
> after a while. I was looking at the name node log and see many of these
> kinds of entries.. Can anything be interpreted from these?
>
> 2013-07-12 09:06:11,256 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 561
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
> 2013-07-12 09:07:11,291 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 14
> 2013-07-12 09:07:11,292 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 15
> 2013-07-12 09:07:11,293 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> 2013-07-12 09:07:11,294 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 563
> 2013-07-12 09:08:11,397 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
> 2013-07-12 09:08:11,399 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 11
> 2013-07-12 09:08:11,400 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 13
> 2013-07-12 09:09:11,441 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 13
>
>
> And after sometime it said:
>
> 2013-07-12 11:03:19,799 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 795
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 12
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 797
> 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:26,003 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Entering safe mode.
> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ON.
> Resources are low on NN. Please add or free up more resources then turn
> off safe mode manually.  NOTE:  If you turn off safe mode before adding
> resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:31,004 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:31,005 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Already in safe mode.
> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ONResources are low on NN. Please add or free up more
> resources then turn off safe mode manually.  NOTE:  If you turn off safe
> mode before adding resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:36,005 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
>
> Thanks,
> Kishore
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Azuryy Yu <az...@gmail.com>.

hi,
from the log:
NameNode low on available disk space. Entering safe mode.

this is the root cause.
 On Jul 15, 2013 2:45 PM, "Krishna Kishore Bonagiri" <
write2kishore@gmail.com> wrote:

> Hi,
>
>  I am doing no activity on my single node cluster which is using
> 2.1.0-beta, and still observed that it has gone to safe mode by itself
> after a while. I was looking at the name node log and see many of these
> kinds of entries.. Can anything be interpreted from these?
>
> 2013-07-12 09:06:11,256 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 561
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
> 2013-07-12 09:07:11,291 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 14
> 2013-07-12 09:07:11,292 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 15
> 2013-07-12 09:07:11,293 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> 2013-07-12 09:07:11,294 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 563
> 2013-07-12 09:08:11,397 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
> 2013-07-12 09:08:11,399 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 11
> 2013-07-12 09:08:11,400 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 13
> 2013-07-12 09:09:11,441 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 13
>
>
> And after sometime it said:
>
> 2013-07-12 11:03:19,799 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 795
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 12
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 797
> 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:26,003 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Entering safe mode.
> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ON.
> Resources are low on NN. Please add or free up more resources then turn
> off safe mode manually.  NOTE:  If you turn off safe mode before adding
> resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:31,004 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:31,005 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Already in safe mode.
> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ONResources are low on NN. Please add or free up more
> resources then turn off safe mode manually.  NOTE:  If you turn off safe
> mode before adding resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:36,005 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
>
> Thanks,
> Kishore
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Venkatarami Netla <ve...@cloudwick.com>.

Hi,
pls see the available space for NN storage directory.

Thanks & Regards

Venkat


On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi,
>
>  I am doing no activity on my single node cluster which is using
> 2.1.0-beta, and still observed that it has gone to safe mode by itself
> after a while. I was looking at the name node log and see many of these
> kinds of entries.. Can anything be interpreted from these?
>
> 2013-07-12 09:06:11,256 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 561
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
> 2013-07-12 09:07:11,291 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 14
> 2013-07-12 09:07:11,292 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 15
> 2013-07-12 09:07:11,293 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> 2013-07-12 09:07:11,294 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 563
> 2013-07-12 09:08:11,397 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
> 2013-07-12 09:08:11,399 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 11
> 2013-07-12 09:08:11,400 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 13
> 2013-07-12 09:09:11,441 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 13
>
>
> And after sometime it said:
>
> 2013-07-12 11:03:19,799 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 795
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 12
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 797
> 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:26,003 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Entering safe mode.
> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ON.
> Resources are low on NN. Please add or free up more resources then turn
> off safe mode manually.  NOTE:  If you turn off safe mode before adding
> resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:31,004 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:31,005 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Already in safe mode.
> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ONResources are low on NN. Please add or free up more
> resources then turn off safe mode manually.  NOTE:  If you turn off safe
> mode before adding resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:36,005 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
>
> Thanks,
> Kishore
>



-- 
N Venkata Rami Reddy
Hadoop Admin
Cloudwick Technologies

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Azuryy Yu <az...@gmail.com>.

hi,
from the log:
NameNode low on available disk space. Entering safe mode.

this is the root cause.
 On Jul 15, 2013 2:45 PM, "Krishna Kishore Bonagiri" <
write2kishore@gmail.com> wrote:

> Hi,
>
>  I am doing no activity on my single node cluster which is using
> 2.1.0-beta, and still observed that it has gone to safe mode by itself
> after a while. I was looking at the name node log and see many of these
> kinds of entries.. Can anything be interpreted from these?
>
> 2013-07-12 09:06:11,256 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 561
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
> 2013-07-12 09:07:11,291 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 14
> 2013-07-12 09:07:11,292 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 15
> 2013-07-12 09:07:11,293 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> 2013-07-12 09:07:11,294 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 563
> 2013-07-12 09:08:11,397 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
> 2013-07-12 09:08:11,399 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 11
> 2013-07-12 09:08:11,400 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 13
> 2013-07-12 09:09:11,441 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 13
>
>
> And after sometime it said:
>
> 2013-07-12 11:03:19,799 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 795
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 12
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 797
> 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:26,003 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Entering safe mode.
> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ON.
> Resources are low on NN. Please add or free up more resources then turn
> off safe mode manually.  NOTE:  If you turn off safe mode before adding
> resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:31,004 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:31,005 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Already in safe mode.
> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ONResources are low on NN. Please add or free up more
> resources then turn off safe mode manually.  NOTE:  If you turn off safe
> mode before adding resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:36,005 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
>
> Thanks,
> Kishore
>

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Venkatarami Netla <ve...@cloudwick.com>.

Hi,
pls see the available space for NN storage directory.

Thanks & Regards

Venkat


On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi,
>
>  I am doing no activity on my single node cluster which is using
> 2.1.0-beta, and still observed that it has gone to safe mode by itself
> after a while. I was looking at the name node log and see many of these
> kinds of entries.. Can anything be interpreted from these?
>
> 2013-07-12 09:06:11,256 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 561
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
> 2013-07-12 09:07:11,291 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 14
> 2013-07-12 09:07:11,292 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 15
> 2013-07-12 09:07:11,293 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> 2013-07-12 09:07:11,294 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 563
> 2013-07-12 09:08:11,397 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
> 2013-07-12 09:08:11,399 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 11
> 2013-07-12 09:08:11,400 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 13
> 2013-07-12 09:09:11,441 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 13
>
>
> And after sometime it said:
>
> 2013-07-12 11:03:19,799 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 795
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 12
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 797
> 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:26,003 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Entering safe mode.
> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ON.
> Resources are low on NN. Please add or free up more resources then turn
> off safe mode manually.  NOTE:  If you turn off safe mode before adding
> resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:31,004 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:31,005 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Already in safe mode.
> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ONResources are low on NN. Please add or free up more
> resources then turn off safe mode manually.  NOTE:  If you turn off safe
> mode before adding resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:36,005 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
>
> Thanks,
> Kishore
>



-- 
N Venkata Rami Reddy
Hadoop Admin
Cloudwick Technologies

Re: Namenode automatically going to safemode with 2.1.0-beta

Posted by Venkatarami Netla <ve...@cloudwick.com>.

Hi,
pls see the available space for NN storage directory.

Thanks & Regards

Venkat


On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi,
>
>  I am doing no activity on my single node cluster which is using
> 2.1.0-beta, and still observed that it has gone to safe mode by itself
> after a while. I was looking at the name node log and see many of these
> kinds of entries.. Can anything be interpreted from these?
>
> 2013-07-12 09:06:11,256 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 561
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:07:11,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 561
> 2013-07-12 09:07:11,291 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 14
> 2013-07-12 09:07:11,292 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 1 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 15
> 2013-07-12 09:07:11,293 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562
> 2013-07-12 09:07:11,294 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 563
> 2013-07-12 09:08:11,397 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:08:11,398 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 563
> 2013-07-12 09:08:11,399 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 11
> 2013-07-12 09:08:11,400 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 2 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564
> 2013-07-12 09:08:11,402 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 565
> 2013-07-12 09:09:11,440 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 13
> 2013-07-12 09:09:11,441 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 13
>
>
> And after sometime it said:
>
> 2013-07-12 11:03:19,799 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 795
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 9.70.137.114
> 2013-07-12 11:04:19,826 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 795
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 2 SyncTimes(ms): 12
> 2013-07-12 11:04:19,827 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
> Total time for transactions(ms): 0 Number of transactions batched in Syncs:
> 0 Number of syncs: 3 SyncTimes(ms): 12
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 ->
> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796
> 2013-07-12 11:04:19,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
> 797
> 2013-07-12 11:04:26,002 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:26,003 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Entering safe mode.
> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ON.
> Resources are low on NN. Please add or free up more resources then turn
> off safe mode manually.  NOTE:  If you turn off safe mode before adding
> resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:31,004 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
> 2013-07-12 11:04:31,005 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
> available disk space. Already in safe mode.
> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode is ONResources are low on NN. Please add or free up more
> resources then turn off safe mode manually.  NOTE:  If you turn off safe
> mode before adding resources, the NN will immediately return to safe mode..
> 2013-07-12 11:04:36,005 WARN
> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
> available on volume 'null' is 0, which is below the configured reserved
> amount 104857600
>
>
> Thanks,
> Kishore
>



-- 
N Venkata Rami Reddy
Hadoop Admin
Cloudwick Technologies