You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Matthew Foley <ma...@yahoo-inc.com> on 2011/04/08 21:27:06 UTC
Re: HDFS start-up with safe mode?
1.. Can I regain percentage of blocks that should satisfy the minimal replication requirement
to 99.9%? hadoop balancer? For I feel it will be more safe.
2. I set "dfs.safemode.threshold.pct" to "0" or "0f", two value both work, but which one is
better? I guess "0"
3. When HDFS start up in safe mode, the log file should show
"The reported blocks 0 needs additional 2 blocks to reach the threshold 0.9990 of total blocks 3. Safe mode will 'not' be turned off automatically."
There miss a word "not" , right?
Regarding your follow-on questions, in case it wasn't clear from my first message (copied below):
At startup time, the namenode reads its namespace from disk (the FSImage and edits files). This includes all the HDFS filenames and block lists that it should know, but not the mappings of block replicas to datanodes. Then it waits in safe mode for all or most of the datanodes to send their Initial Block Reports, which let the namenode build its map of which blocks have replicas in which datanodes. It keeps waiting until dfs.namenode.safemode.threshold-pct of the blocks that it knows about from FSImage have been reported from at least dfs.namenode.replication.min (default 1) datanodes [so that's a third config parameter I didn't mention earlier]. If this threshold is achieved, it will post a log that it is ready to leave safe mode, wait for dfs.namenode.safemode.extension seconds, then automatically leave safe mode and generate replication requests for any under-replicated blocks (by default, those with replication < 3).
If it doesn't reach the "safe replication for all known blocks" threshold, then it will not leave safe mode automatically. It logs the condition and waits for an admin to decide what to do, because generally it means whole datanodes or sets of datanodes did not come up or are not able to communicate with the namenode. Hadoop wants a human to look at the situation before hadoop starts trying to madly generate re-replication commands for under-replicated blocks, and deleting blocks with zero replicas available.
BTW, this thread is better for hdfs-user than hdfs-dev.
--Matt
On Apr 8, 2011, at 11:52 AM, Matthew Foley wrote:
From: Matthew Foley <ma...@yahoo-inc.com>>
Date: April 8, 2011 11:52:00 AM PDT
To: "common-user@hadoop.apache.org<ma...@hadoop.apache.org>" <co...@hadoop.apache.org>>
Cc: Matthew Foley <ma...@yahoo-inc.com>>
Subject: Re: start-up with safe mode?
Hi Ring,
The purpose of starting up with safe mode enabled, is to prevent replication thrashing before and during Initial Block Reports from all the datanodes. Consider this thought experiment:
- a cluster with 100 datanodes and replication 3
- so any pair of datanodes only have aprx 2% overlap in their block content
- the datanodes don't all start up exactly simultaneously
- When the first two datanodes start up, if the cluster weren't in safe mode, 98% of their blocks would be declared under-replicated, and they would immediately be asked to replicate them ALL to each other!
- When the third datanode starts up, it gets even worse, since it also only has a 2% overlap with each of the other two.
- It just continues getting worse until many of the datanodes are registered, and the rate slows down for introduction of new blocks with only one found replica.
While safe mode is on, the namenode doesn't attempt to change anything in the namespace or blockspace, including which datanodes have replicas of which blocks, although it does accept Block Reports from the datanodes telling which blocks they have. So the above described replication storm doesn't happen during safe mode. All (or almost all) the datanodes are allowed to register and give their Block Reports. THEN the namenode scans for blocks that truly are under-replicated. It gives a 30-second warning, then leaves safe mode, and generates appropriate replication requests to fix any under-replicated blocks now known.
That said, you can modify this behavior with the configuration parameters
dfs.namenode.safemode.threshold-pct
and dfs.namenode.safemode.extension
These default to 0.999 (100% minus delta), and 30000 (30 sec), respectively (defined in DFSConfigKeys).
Search for them in the docs for details.
If you want non-default values, you'd typically set them in hdfs-site.xml in your namenode's config directory.
Setting them both to 0 will give you what you are asking for, but it probably isn't what you want :-)
--Matt
On Apr 8, 2011, at 2:36 AM, springring wrote:
I modify the value of "dfs.safemode.threshold.pct" to zero, now everything is ok.
log file as below
But there are still three questions
1.. Can I regain percentage of blocks that should satisfy the minimal replication requirement
to 99.9%? hadoop balancer? For I feel it will be more safe.
2. I set "dfs.safemode.threshold.pct" to "0" or "0f", two value both work, but which one is
better? I guess "0"
3. When HDFS start up in safe mode, the log file should show
"The reported blocks 0 needs additional 2 blocks to reach the threshold 0.9990 of total blocks 3. Safe mode will 'not' be turned off automatically."
There miss a word "not" , right?
Ring
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at computeb-05.pcm/172.172.2.6
************************************************************/
2011-04-08 16:33:37,312 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = computeb-05.pcm/172.172.2.6
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2-CDH3B4
STARTUP_MSG: build = -r 3aa7c91592ea1c53f3a913a581dbfcdfebe98bfe; compiled by 'root' on Mon Feb 21 17:31:12 EST 2011
************************************************************/
2011-04-08 16:33:37,441 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null
2011-04-08 16:33:37,443 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext
2011-04-08 16:33:37,464 INFO org.apache.hadoop.hdfs.util.GSet: VM type = 32-bit
2011-04-08 16:........................................
2011-04-08 16:33:37,832 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of under-replicated blocks = 4
2011-04-08 16:33:37,832 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of over-replicated blocks = 0
2011-04-08 16:33:37,832 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 0 secs.
2011-04-08 16:33:37,832 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 0 racks and 0 datanodes
2011-04-08 16:33:37,832 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 4 blocks
2011-04-08 16:33:37,835 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2011-04-08 16:33:37,849 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 9100
----- Original Message -----
From: "springring" <sp...@126.com>>
To: <co...@hadoop.apache.org>>
Cc: <hd...@hadoop.apache.org>>
Sent: Friday, April 08, 2011 3:45 PM
Subject: Re:HDFS start-up with safe mode?
Hi,
I guess that something about "threshold 0.9990". When HDFS start up,
it come in safe mode first, then check a value(I don't know what value or percent?)
of my hadoop,and fine the value below 99.9%, so the safe mode will not turn off?
but the conclusion of the log file is "Safe mode will be turned off automatically"?
I'm lost.
___________________________________________________
2011-04-08 11:58:21,036 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 0 needs additional 2 blocks to reach the threshold 0.9990 of total blocks 3. Safe mode will be turned off automatically.
________________________________________________________________________
----- Original Message -----
From: "springring" <sp...@126.com>>
To: <co...@hadoop.apache.org>>
Sent: Friday, April 08, 2011 2:20 PM
Subject: Fw: start-up with safe mode?
Hi,
When I start up hadoop, the namenode log show "STATE* Safe mode ON" like that , how to set it off?
I can set it off with command "hadoop fs -dfsadmin leave" after start up, but how can I just start HDFS
out of Safe mode?
Thanks.
Ring
the startup log________________________________________________________________
2011-04-08 11:58:20,655 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null
2011-04-08 11:58:20,657 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext
2011-04-08 11:58:20,678 INFO org.apache.hadoop.hdfs.util.GSet: VM type = 32-bit
2011-04-08 11:58:20,678 INFO org.apache.hadoop.hdfs.util.GSet: 2% max memory = 17.77875 MB
2011-04-08 11:58:20,678 INFO org.apache.hadoop.hdfs.util.GSet: capacity = 2^22 = 4194304 entries
2011-04-08 11:58:20,678 INFO org.apache.hadoop.hdfs.util.GSet: recommended=4194304, actual=4194304
2011-04-08 11:58:20,697 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=hdfs
2011-04-08 11:58:20,697 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
2011-04-08 11:58:20,697 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true
2011-04-08 11:58:20,701 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.block.invalidate.limit=1000
2011-04-08 11:58:20,701 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
2011-04-08 11:58:20,976 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context object:org.apache.hadoop.metrics.spi.NullContext
2011-04-08 11:58:21,001 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 17
2011-04-08 11:58:21,007 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 0
2011-04-08 11:58:21,007 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 1529 loaded in 0 seconds.
2011-04-08 11:58:21,007 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /tmp/hadoop-hdfs/dfs/name/current/edits of size 4 edits # 0 loaded in 0 seconds.
2011-04-08 11:58:21,009 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 1529 saved in 0 seconds.
2011-04-08 11:58:21,022 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 1529 saved in 0 seconds.
2011-04-08 11:58:21,032 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 339 msecs
2011-04-08 11:58:21,036 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 0 needs additional 2 blocks to reach the threshold 0.9990 of total blocks 3. Safe mode will be turned off automatically.