You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "suja s (Created) (JIRA)" <ji...@apache.org> on 2012/03/29 10:52:30 UTC
[jira] [Created] (HDFS-3161) 20 Append: Excluded DN replica from
recovery should be removed from DN.
20 Append: Excluded DN replica from recovery should be removed from DN.
-----------------------------------------------------------------------
Key: HDFS-3161
URL: https://issues.apache.org/jira/browse/HDFS-3161
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: suja s
Priority: Critical
Fix For: 1.0.3
1) DN1->DN2->DN3 are in pipeline.
2) Client killed abruptly
3) one DN has restarted , say DN3
4) In DN3 info.wasRecoveredOnStartup() will be true
5) NN recovery triggered, DN3 skipped from recovery due to above check.
6) Now DN1, DN2 has blocks with generataion stamp 2 and DN3 has older generation stamp say 1 and also DN3 still has this block entry in ongoingCreates
7) as part of recovery file has closed and got only two live replicas ( from DN1 and DN2)
8) So, NN issued the command for replication. Now DN3 also has the replica with newer generation stamp.
9) Now DN3 contains 2 replicas on disk. and one entry in ongoing creates with referring to blocksBeingWritten directory.
When we call append/ leaseRecovery, it may again skip this node for that recovery as blockId entry still presents in ongoingCreates with startup recovery true.
It may keep continue this dance for evry recovery.
And this stale replica will not be cleaned untill we restart the cluster. Actual replica will be trasferred to this node only through replication process.
Also unnecessarily that replicated blocks will get invalidated after next recoveries....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3161) 20 Append: Excluded DN replica from
recovery should be removed from DN.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241105#comment-13241105 ]
Uma Maheswara Rao G commented on HDFS-3161:
-------------------------------------------
Here actually DN3 has removed from the pipeline while recovery. But unfortunately replication block again went to DN3. So, the problem start as the entry presents in ongoingcreates. If the replication block goes to other node then, I think problem may not come.
One quick thought,
can't we remove onGoingCreates entry on TRANSFER_BLOCK command? Because, replication triggered means, that block must have been completed. So, ideally there should not be any entry in onGoingCreates. But only the problem is that stale block may not be cleaned until we restart the cluster. But that won't create any problem for further pipelines.
Please suggest if you have better solution.
> 20 Append: Excluded DN replica from recovery should be removed from DN.
> -----------------------------------------------------------------------
>
> Key: HDFS-3161
> URL: https://issues.apache.org/jira/browse/HDFS-3161
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: suja s
> Priority: Critical
> Fix For: 1.0.3
>
>
> 1) DN1->DN2->DN3 are in pipeline.
> 2) Client killed abruptly
> 3) one DN has restarted , say DN3
> 4) In DN3 info.wasRecoveredOnStartup() will be true
> 5) NN recovery triggered, DN3 skipped from recovery due to above check.
> 6) Now DN1, DN2 has blocks with generataion stamp 2 and DN3 has older generation stamp say 1 and also DN3 still has this block entry in ongoingCreates
> 7) as part of recovery file has closed and got only two live replicas ( from DN1 and DN2)
> 8) So, NN issued the command for replication. Now DN3 also has the replica with newer generation stamp.
> 9) Now DN3 contains 2 replicas on disk. and one entry in ongoing creates with referring to blocksBeingWritten directory.
> When we call append/ leaseRecovery, it may again skip this node for that recovery as blockId entry still presents in ongoingCreates with startup recovery true.
> It may keep continue this dance for evry recovery.
> And this stale replica will not be cleaned untill we restart the cluster. Actual replica will be trasferred to this node only through replication process.
> Also unnecessarily that replicated blocks will get invalidated after next recoveries....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3161) 20 Append: Excluded DN replica from
recovery should be removed from DN.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241095#comment-13241095 ]
Uma Maheswara Rao G commented on HDFS-3161:
-------------------------------------------
Thanks a lot Suja for reporting the issue.
> 20 Append: Excluded DN replica from recovery should be removed from DN.
> -----------------------------------------------------------------------
>
> Key: HDFS-3161
> URL: https://issues.apache.org/jira/browse/HDFS-3161
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: suja s
> Priority: Critical
> Fix For: 1.0.3
>
>
> 1) DN1->DN2->DN3 are in pipeline.
> 2) Client killed abruptly
> 3) one DN has restarted , say DN3
> 4) In DN3 info.wasRecoveredOnStartup() will be true
> 5) NN recovery triggered, DN3 skipped from recovery due to above check.
> 6) Now DN1, DN2 has blocks with generataion stamp 2 and DN3 has older generation stamp say 1 and also DN3 still has this block entry in ongoingCreates
> 7) as part of recovery file has closed and got only two live replicas ( from DN1 and DN2)
> 8) So, NN issued the command for replication. Now DN3 also has the replica with newer generation stamp.
> 9) Now DN3 contains 2 replicas on disk. and one entry in ongoing creates with referring to blocksBeingWritten directory.
> When we call append/ leaseRecovery, it may again skip this node for that recovery as blockId entry still presents in ongoingCreates with startup recovery true.
> It may keep continue this dance for evry recovery.
> And this stale replica will not be cleaned untill we restart the cluster. Actual replica will be trasferred to this node only through replication process.
> Also unnecessarily that replicated blocks will get invalidated after next recoveries....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3161) 20 Append: Excluded DN replica from
recovery should be removed from DN.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248247#comment-13248247 ]
Uma Maheswara Rao G commented on HDFS-3161:
-------------------------------------------
Yes, I am seeing many issues related 20-Append, all are mainly due to recovery.
I agree with you (also i have seen many times you have pointed it), since append is working pretty well in latest releases than 20-append, we can disable it.
Since we are dropping support for 20 append issues, it is better, we can officially announce?
Some customers have started using this append unknowingly.
> 20 Append: Excluded DN replica from recovery should be removed from DN.
> -----------------------------------------------------------------------
>
> Key: HDFS-3161
> URL: https://issues.apache.org/jira/browse/HDFS-3161
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: suja s
> Priority: Critical
> Fix For: 1.0.3
>
>
> 1) DN1->DN2->DN3 are in pipeline.
> 2) Client killed abruptly
> 3) one DN has restarted , say DN3
> 4) In DN3 info.wasRecoveredOnStartup() will be true
> 5) NN recovery triggered, DN3 skipped from recovery due to above check.
> 6) Now DN1, DN2 has blocks with generataion stamp 2 and DN3 has older generation stamp say 1 and also DN3 still has this block entry in ongoingCreates
> 7) as part of recovery file has closed and got only two live replicas ( from DN1 and DN2)
> 8) So, NN issued the command for replication. Now DN3 also has the replica with newer generation stamp.
> 9) Now DN3 contains 2 replicas on disk. and one entry in ongoing creates with referring to blocksBeingWritten directory.
> When we call append/ leaseRecovery, it may again skip this node for that recovery as blockId entry still presents in ongoingCreates with startup recovery true.
> It may keep continue this dance for evry recovery.
> And this stale replica will not be cleaned untill we restart the cluster. Actual replica will be trasferred to this node only through replication process.
> Also unnecessarily that replicated blocks will get invalidated after next recoveries....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3161) 20 Append: Excluded DN replica from
recovery should be removed from DN.
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255026#comment-13255026 ]
Todd Lipcon commented on HDFS-3161:
-----------------------------------
Hi Uma/Vinay.
I ran into an issue like this without use of append():
- Client writing blk_N_GS1 to DN1, DN9, DN10
- Pipeline failed. commitBlockSynchronization succeeded with DN9 and DN10, sets gs to blk_N_GS2
- Client closes the pipeline
- NN issues replication request of blk_N_GS2 from DN9 to DN1
- DN1 already has blk_N_GS1 in its ongoingCreates map
I'm not sure if this can cause any serious issue with the block (it didn't in my case), but I agree that, if a replication request happens for a block with a higher genstamp, it should interrupt the old block's ongoingCreate. If the replication request is a lower genstamp, it should be ignored.
> 20 Append: Excluded DN replica from recovery should be removed from DN.
> -----------------------------------------------------------------------
>
> Key: HDFS-3161
> URL: https://issues.apache.org/jira/browse/HDFS-3161
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: suja s
> Priority: Critical
> Fix For: 1.0.3
>
>
> 1) DN1->DN2->DN3 are in pipeline.
> 2) Client killed abruptly
> 3) one DN has restarted , say DN3
> 4) In DN3 info.wasRecoveredOnStartup() will be true
> 5) NN recovery triggered, DN3 skipped from recovery due to above check.
> 6) Now DN1, DN2 has blocks with generataion stamp 2 and DN3 has older generation stamp say 1 and also DN3 still has this block entry in ongoingCreates
> 7) as part of recovery file has closed and got only two live replicas ( from DN1 and DN2)
> 8) So, NN issued the command for replication. Now DN3 also has the replica with newer generation stamp.
> 9) Now DN3 contains 2 replicas on disk. and one entry in ongoing creates with referring to blocksBeingWritten directory.
> When we call append/ leaseRecovery, it may again skip this node for that recovery as blockId entry still presents in ongoingCreates with startup recovery true.
> It may keep continue this dance for evry recovery.
> And this stale replica will not be cleaned untill we restart the cluster. Actual replica will be trasferred to this node only through replication process.
> Also unnecessarily that replicated blocks will get invalidated after next recoveries....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3161) 20 Append: Excluded DN replica from
recovery should be removed from DN.
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HDFS-3161:
-----------------------------
Target Version/s: 1.1.0 (was: 1.0.3)
Fix Version/s: (was: 1.0.3)
No patch submitted in 1.0.3 timeframe. If still intend to fix, please work in 1.1 context (branch-1). Thank you.
> 20 Append: Excluded DN replica from recovery should be removed from DN.
> -----------------------------------------------------------------------
>
> Key: HDFS-3161
> URL: https://issues.apache.org/jira/browse/HDFS-3161
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: suja s
> Priority: Critical
>
> 1) DN1->DN2->DN3 are in pipeline.
> 2) Client killed abruptly
> 3) one DN has restarted , say DN3
> 4) In DN3 info.wasRecoveredOnStartup() will be true
> 5) NN recovery triggered, DN3 skipped from recovery due to above check.
> 6) Now DN1, DN2 has blocks with generataion stamp 2 and DN3 has older generation stamp say 1 and also DN3 still has this block entry in ongoingCreates
> 7) as part of recovery file has closed and got only two live replicas ( from DN1 and DN2)
> 8) So, NN issued the command for replication. Now DN3 also has the replica with newer generation stamp.
> 9) Now DN3 contains 2 replicas on disk. and one entry in ongoing creates with referring to blocksBeingWritten directory.
> When we call append/ leaseRecovery, it may again skip this node for that recovery as blockId entry still presents in ongoingCreates with startup recovery true.
> It may keep continue this dance for evry recovery.
> And this stale replica will not be cleaned untill we restart the cluster. Actual replica will be trasferred to this node only through replication process.
> Also unnecessarily that replicated blocks will get invalidated after next recoveries....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3161) 20 Append: Excluded DN replica from
recovery should be removed from DN.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255269#comment-13255269 ]
Uma Maheswara Rao G commented on HDFS-3161:
-------------------------------------------
Todd, This situation can occur, but we have seen the problem only in append, as it will skip the recoveries if entry presents in ongoingCreates.
In this case:
bq. DN1 already has blk_N_GS1 in its ongoingCreates map
Block transfer will happen successfully as part of replication. Also reader should be able to read it with newer genstamp. If there is no append for this file, then there won't be any recoveries. So, we need not worry about skipping the DN from recovery if it presents in ongoing creates.
Only thing is, that block and ongoing creates entry will not be cleared until we restart the cluster.
Anyway let me confirm whether readers are able to read properly or not. And will write a test with your scenario.
Do you think any other problems?
{quote}
but I agree that, if a replication request happens for a block with a higher genstamp, it should interrupt the old block's ongoingCreate. If the replication request is a lower genstamp, it should be ignored.
{quote}
If we really want to address this case, then this would be the fix I feel.
> 20 Append: Excluded DN replica from recovery should be removed from DN.
> -----------------------------------------------------------------------
>
> Key: HDFS-3161
> URL: https://issues.apache.org/jira/browse/HDFS-3161
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: suja s
> Priority: Critical
> Fix For: 1.0.3
>
>
> 1) DN1->DN2->DN3 are in pipeline.
> 2) Client killed abruptly
> 3) one DN has restarted , say DN3
> 4) In DN3 info.wasRecoveredOnStartup() will be true
> 5) NN recovery triggered, DN3 skipped from recovery due to above check.
> 6) Now DN1, DN2 has blocks with generataion stamp 2 and DN3 has older generation stamp say 1 and also DN3 still has this block entry in ongoingCreates
> 7) as part of recovery file has closed and got only two live replicas ( from DN1 and DN2)
> 8) So, NN issued the command for replication. Now DN3 also has the replica with newer generation stamp.
> 9) Now DN3 contains 2 replicas on disk. and one entry in ongoing creates with referring to blocksBeingWritten directory.
> When we call append/ leaseRecovery, it may again skip this node for that recovery as blockId entry still presents in ongoingCreates with startup recovery true.
> It may keep continue this dance for evry recovery.
> And this stale replica will not be cleaned untill we restart the cluster. Actual replica will be trasferred to this node only through replication process.
> Also unnecessarily that replicated blocks will get invalidated after next recoveries....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3161) 20 Append: Excluded DN replica from
recovery should be removed from DN.
Posted by "VinayaKumar B (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247102#comment-13247102 ]
VinayaKumar B commented on HDFS-3161:
-------------------------------------
Hi Uma,
One more scenario, where same block with different genstamps can be present at DN.
Scenario 2: Both blocks in current
---------------------------------
1) File is written to DN1-> DN2 -> DN3, and file closed with generation stamp 1.
2) Now DN3 network down.
3) Append is called on same file, and append recovery is success in DN1 and DN2 with genstamp 2.
4) Some more data is written and append stream is closed with genstamp 2.
5) Now DN3 network comes back.
6) Now before DN3 sends the block report to NN, NN asked for replication of block with genstamp 2 to DN3.
7) After Replication DN3 will have same block with 2 genstamps.
8) On Next Block Report NN will invalidate the block with genstamp 1.
Here, if the subdir of both blocks is same, then invalidation of old block, will delete the actual block file, resulting scan failure.
In the next block report, NN will remove this datanode from the blocksMap, and again replication will happen. But for replication another DN is choosen then, this old block related entries will be still present in DN memory.
> 20 Append: Excluded DN replica from recovery should be removed from DN.
> -----------------------------------------------------------------------
>
> Key: HDFS-3161
> URL: https://issues.apache.org/jira/browse/HDFS-3161
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: suja s
> Priority: Critical
> Fix For: 1.0.3
>
>
> 1) DN1->DN2->DN3 are in pipeline.
> 2) Client killed abruptly
> 3) one DN has restarted , say DN3
> 4) In DN3 info.wasRecoveredOnStartup() will be true
> 5) NN recovery triggered, DN3 skipped from recovery due to above check.
> 6) Now DN1, DN2 has blocks with generataion stamp 2 and DN3 has older generation stamp say 1 and also DN3 still has this block entry in ongoingCreates
> 7) as part of recovery file has closed and got only two live replicas ( from DN1 and DN2)
> 8) So, NN issued the command for replication. Now DN3 also has the replica with newer generation stamp.
> 9) Now DN3 contains 2 replicas on disk. and one entry in ongoing creates with referring to blocksBeingWritten directory.
> When we call append/ leaseRecovery, it may again skip this node for that recovery as blockId entry still presents in ongoingCreates with startup recovery true.
> It may keep continue this dance for evry recovery.
> And this stale replica will not be cleaned untill we restart the cluster. Actual replica will be trasferred to this node only through replication process.
> Also unnecessarily that replicated blocks will get invalidated after next recoveries....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3161) 20 Append: Excluded DN replica from
recovery should be removed from DN.
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247446#comment-13247446 ]
Todd Lipcon commented on HDFS-3161:
-----------------------------------
If this is 0.20-append specific, we've recently decided to disable the append() call in that branch (and only support sync()). So, I don't think the append-related scenario is worth worrying about (assuming it works correctly in the trunk implementation)
> 20 Append: Excluded DN replica from recovery should be removed from DN.
> -----------------------------------------------------------------------
>
> Key: HDFS-3161
> URL: https://issues.apache.org/jira/browse/HDFS-3161
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: suja s
> Priority: Critical
> Fix For: 1.0.3
>
>
> 1) DN1->DN2->DN3 are in pipeline.
> 2) Client killed abruptly
> 3) one DN has restarted , say DN3
> 4) In DN3 info.wasRecoveredOnStartup() will be true
> 5) NN recovery triggered, DN3 skipped from recovery due to above check.
> 6) Now DN1, DN2 has blocks with generataion stamp 2 and DN3 has older generation stamp say 1 and also DN3 still has this block entry in ongoingCreates
> 7) as part of recovery file has closed and got only two live replicas ( from DN1 and DN2)
> 8) So, NN issued the command for replication. Now DN3 also has the replica with newer generation stamp.
> 9) Now DN3 contains 2 replicas on disk. and one entry in ongoing creates with referring to blocksBeingWritten directory.
> When we call append/ leaseRecovery, it may again skip this node for that recovery as blockId entry still presents in ongoingCreates with startup recovery true.
> It may keep continue this dance for evry recovery.
> And this stale replica will not be cleaned untill we restart the cluster. Actual replica will be trasferred to this node only through replication process.
> Also unnecessarily that replicated blocks will get invalidated after next recoveries....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira