You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ludun (Jira)" <ji...@apache.org> on 2020/06/23 09:05:00 UTC
[jira] [Commented] (HDFS-15431) Can not read a opening file after
NameNode failover if pipeline recover occuered
[ https://issues.apache.org/jira/browse/HDFS-15431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17142764#comment-17142764 ]
ludun commented on HDFS-15431:
------------------------------
In BlockReceiver we should notifyNamenodeReceivingBlock also when PIPELINE_SETUP_STREAMING_RECOVERY
{code}
case PIPELINE_SETUP_CREATE:
replicaHandler = datanode.data.createRbw(storageType, storageId,
block, allowLazyPersist);
datanode.notifyNamenodeReceivingBlock(
block, replicaHandler.getReplica().getStorageUuid());
break;
case PIPELINE_SETUP_STREAMING_RECOVERY:
replicaHandler = datanode.data.recoverRbw(
block, newGs, minBytesRcvd, maxBytesRcvd);
block.setGenerationStamp(newGs);
break;
{code}
so standby namenode can also know the location of new dn. after failerover we can read file normal.
> Can not read a opening file after NameNode failover if pipeline recover occuered
> --------------------------------------------------------------------------------
>
> Key: HDFS-15431
> URL: https://issues.apache.org/jira/browse/HDFS-15431
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Reporter: ludun
> Priority: Major
>
> a file with two replications and keep it opening.
> first it writes to DN1 and DN2.
> {code}
> 2020-06-23 14:22:51,379 | DEBUG | pipeline = [DatanodeInfoWithStorage[DN1:25009,DS-1dcbe5bd-f69a-422c-bea6-a41bda773084,DISK], DatanodeInfoWithStorage[DN2:25009,DS-7e434b35-0b10-44fa-9d3b-c3c938f1724d,DISK]] | DataStreamer.java:1757
> {code}
> after DN2 restart, it writes to DN1 and DN3,
> {code}
> 2020-06-23 14:24:04,559 | DEBUG | pipeline = [DatanodeInfoWithStorage[DN1:25009,DS-1dcbe5bd-f69a-422c-bea6-a41bda773084,DISK], DatanodeInfoWithStorage[DN3:25009,DS-1810c3d5-b6e8-4403-a0fc-071ea6e5489f,DISK]] | DataStreamer.java:1757
> {code}
> after DN1 restart. it writes to DN3 and DN4.
> {code}
> 2020-06-23 14:25:21,340 | DEBUG | pipeline = [DatanodeInfoWithStorage[DN3:25009,DS-1810c3d5-b6e8-4403-a0fc-071ea6e5489f,DISK], DatanodeInfoWithStorage[DN4:25009,DS-5fbb2232-e7c8-4186-8eb9-87a6aff86cef,DISK]] | DataStreamer.java:1757
> {code}
> restart Active NameNode. then try to get the file.
> NameNode return locatedblocks with DN1 and DN2. Can not obtain block Exception occurred.
> {code}
> 20/06/20 17:57:06 DEBUG hdfs.DFSClient: newInfo = LocatedBlocks{
> fileLength=0
> underConstruction=true
> blocks=[LocatedBlock{BP-1590194288-10.162.26.113-1587096223927:blk_1073895975_155796; getBlockSize()=53; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[DN1:25009,DS-1dcbe5bd-f69a-422c-bea6-a41bda773084,DISK], DatanodeInfoWithStorage[DN2:25009,DS-cd06a4f9-c25d-42ab-887b-f129707dba17,DISK]]}]
> lastLocatedBlock=LocatedBlock{BP-1590194288-10.162.26.113-1587096223927:blk_1073895975_155796; getBlockSize()=53; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[DN1:25009,DS-1dcbe5bd-f69a-422c-bea6-a41bda773084,DISK], DatanodeInfoWithStorage[DN2:25009,DS-cd06a4f9-c25d-42ab-887b-f129707dba17,DISK]]}
> isLastBlockComplete=false}
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org