After NameNode restarts, Can not read a file until Datanodes restart

[심탁길 <10...@nhncorp.com>]

 
For example, New file "a.txt" is created on DFS.
 
Before NameNode shuts down, A pair of Block and DataNodeInfo[] for "a.txt" is displayed (Using LocatedBlock class)
 
After NameNode restarts, Only a Block Info remains, DataNodeInfo disappears...
 
So when try to read the file, the message like "No node available for block: blk_XXXXXXX ...... No live nodes contain current block"
 
is displayed.  and then when Datanodes restarts, the problem is solved and Client can access to the file.
 
For Cluster with small Nodes it could be ok to restart all the NameNode and DataNodes together, but Big Cluster more than hundreds of Nodes is different story.
 
Any Comments would be appreciated
 
 
 

Re: After NameNode restarts, Can not read a file until Datanodes restart

[Konstantin Shvachko <sh...@yahoo-inc.com>]

Hi  심탁길

If you restart the name-node only, the data-nodes will continue sending 
block reports to the name-node
once in an hour, which is the default block-report interval. That means 
all blocks will be reported to the
name-node sooner or later. But you want that sooner of course.
IMO we should request block reports when the name-node restarts.
For now I recommend to wait 10 minutes between shutdown and restart. 
Then all data-nodes
will expire and the name-node will ask them to send block reports asap. 
This works with the
trunk version. HADOOP-641 is scheduled for 0.8.0.

Константин


심탁길 wrote:

> 
>For example, New file "a.txt" is created on DFS.
> 
>Before NameNode shuts down, A pair of Block and DataNodeInfo[] for "a.txt" is displayed (Using LocatedBlock class)
> 
>After NameNode restarts, Only a Block Info remains, DataNodeInfo disappears...
> 
>So when try to read the file, the message like "No node available for block: blk_XXXXXXX ...... No live nodes contain current block"
> 
>is displayed.  and then when Datanodes restarts, the problem is solved and Client can access to the file.
> 
>For Cluster with small Nodes it could be ok to restart all the NameNode and DataNodes together, but Big Cluster more than hundreds of Nodes is different story.
> 
>Any Comments would be appreciated
> 
> 
> 
>

RE: After NameNode restarts, Can not read a file until Datanodes restart

[Yoram Arnon <ya...@yahoo-inc.com>]

It's sort-of inherent: until the datanodes start and report in, the data is
unavailable.

Under normal operation, if a datanode dies its data is replicated and all
is well, but during startup, until most of the datanodes start up and
report it the data is not there. As a rule, with a replication factor of 3,
as long as three or more nodes are missing some data will be missing.

Safe mode makes this both clear and configurable: you configure the
percentage of blocks required to exit safe mode, and the time to wait to
allow the last nodes to report. The default is good for large clusters:
99.9% of the blocks plus 30 seconds and yes, it takes a few minutes for a
several hundred node cluster to start. Smaller values may be configured for
a small/toy cluster.

Yoram

> -----Original Message-----
> From: 심탁길 [mailto:1004shi@nhncorp.com] 
> Sent: Monday, October 30, 2006 6:28 PM
> To: hadoop-dev@lucene.apache.org
> Subject: After NameNode restarts, Can not read a file until 
> Datanodes restart
> 
> 
>  
> For example, New file "a.txt" is created on DFS.
>  
> Before NameNode shuts down, A pair of Block and 
> DataNodeInfo[] for "a.txt" is displayed (Using LocatedBlock class)
>  
> After NameNode restarts, Only a Block Info remains, 
> DataNodeInfo disappears...
>  
> So when try to read the file, the message like "No node 
> available for block: blk_XXXXXXX ...... No live nodes contain 
> current block"
>  
> is displayed.  and then when Datanodes restarts, the problem 
> is solved and Client can access to the file.
>  
> For Cluster with small Nodes it could be ok to restart all 
> the NameNode and DataNodes together, but Big Cluster more 
> than hundreds of Nodes is different story.
>  
> Any Comments would be appreciated
>  
>  
>