You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Brahma Reddy Battula (JIRA)" <ji...@apache.org> on 2012/07/02 16:30:40 UTC
[jira] [Created] (HDFS-3586) Blocks are not getting replicate even
DN's are availble.
Brahma Reddy Battula created HDFS-3586:
------------------------------------------
Summary: Blocks are not getting replicate even DN's are availble.
Key: HDFS-3586
URL: https://issues.apache.org/jira/browse/HDFS-3586
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node, name-node
Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 3.0.0
Reporter: Brahma Reddy Battula
Scenario:
=========
Started four DN's(Say DN1,DN2,DN3 and DN4)
writing files with RF=3..
formed pipeline with DN1->DN2->DN3.
Since DN3 network is very slow.it's not able to send acks.
Again pipeline is fromed with DN1->DN2->DN4.
Here DN4 network is also slow.
So finally commitblocksync happend tp DN1 and DN2 successfully.
block present in all the four DN's(finalized state in two DN's and rbw state in another DN's)..
Here NN is asking replicate to DN3 and DN4,but it's failing since replcia's are already present in RBW dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3586) Blocks are not getting replicate
even DN's are availble.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406314#comment-13406314 ]
Konstantin Shvachko commented on HDFS-3586:
-------------------------------------------
So you propose to sacrifice a corrupt replica when good ones are present. That makes sense to me. Useful for small clusters, agreed.
> Blocks are not getting replicate even DN's are availble.
> --------------------------------------------------------
>
> Key: HDFS-3586
> URL: https://issues.apache.org/jira/browse/HDFS-3586
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 3.0.0
> Reporter: Brahma Reddy Battula
> Attachments: HDFS-3586-analysis.txt
>
>
> Scenario:
> =========
> Started four DN's(Say DN1,DN2,DN3 and DN4)
> writing files with RF=3..
> formed pipeline with DN1->DN2->DN3.
> Since DN3 network is very slow.it's not able to send acks.
> Again pipeline is fromed with DN1->DN2->DN4.
> Here DN4 network is also slow.
> So finally commitblocksync happend tp DN1 and DN2 successfully.
> block present in all the four DN's(finalized state in two DN's and rbw state in another DN's)..
> Here NN is asking replicate to DN3 and DN4,but it's failing since replcia's are already present in RBW dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3586) Blocks are not getting replicate even
DN's are availble.
Posted by "Brahma Reddy Battula (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brahma Reddy Battula updated HDFS-3586:
---------------------------------------
Attachment: HDFS-3586-analysis.txt
Attaching anlyased sheet
> Blocks are not getting replicate even DN's are availble.
> --------------------------------------------------------
>
> Key: HDFS-3586
> URL: https://issues.apache.org/jira/browse/HDFS-3586
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 3.0.0
> Reporter: Brahma Reddy Battula
> Attachments: HDFS-3586-analysis.txt
>
>
> Scenario:
> =========
> Started four DN's(Say DN1,DN2,DN3 and DN4)
> writing files with RF=3..
> formed pipeline with DN1->DN2->DN3.
> Since DN3 network is very slow.it's not able to send acks.
> Again pipeline is fromed with DN1->DN2->DN4.
> Here DN4 network is also slow.
> So finally commitblocksync happend tp DN1 and DN2 successfully.
> block present in all the four DN's(finalized state in two DN's and rbw state in another DN's)..
> Here NN is asking replicate to DN3 and DN4,but it's failing since replcia's are already present in RBW dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3586) Blocks are not getting replicate
even DN's are availble.
Posted by "Han Xiao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433835#comment-13433835 ]
Han Xiao commented on HDFS-3586:
--------------------------------
Uma, you are right. In my comments, there also be a condition (no in expression) as "as long as good one exists()".
Your description is exactly coninsiding with my ideas. Hope it will be done.
> Blocks are not getting replicate even DN's are availble.
> --------------------------------------------------------
>
> Key: HDFS-3586
> URL: https://issues.apache.org/jira/browse/HDFS-3586
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 2.0.0-alpha, 2.1.0-alpha, 3.0.0
> Reporter: Brahma Reddy Battula
> Assignee: amith
> Attachments: HDFS-3586-analysis.txt
>
>
> Scenario:
> =========
> Started four DN's(Say DN1,DN2,DN3 and DN4)
> writing files with RF=3..
> formed pipeline with DN1->DN2->DN3.
> Since DN3 network is very slow.it's not able to send acks.
> Again pipeline is fromed with DN1->DN2->DN4.
> Here DN4 network is also slow.
> So finally commitblocksync happend tp DN1 and DN2 successfully.
> block present in all the four DN's(finalized state in two DN's and rbw state in another DN's)..
> Here NN is asking replicate to DN3 and DN4,but it's failing since replcia's are already present in RBW dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3586) Blocks are not getting replicate
even DN's are availble.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405404#comment-13405404 ]
Konstantin Shvachko commented on HDFS-3586:
-------------------------------------------
Uma, I think that the RBW replicas on DN3 and DN4 should eventually be transformed into corrupt replicas, because they should have a wrong generation stamp.
Then you are right it is similar to HDFS-3493. There is 4 nodes more than replication 3, but there are no nodes that can be used to replicate the block to, because all have a replica of the block.
I don't see how your suggestion solves the problem, as it doesn't work in case when all replicas are corrupt, if I understand it correctly.
> Blocks are not getting replicate even DN's are availble.
> --------------------------------------------------------
>
> Key: HDFS-3586
> URL: https://issues.apache.org/jira/browse/HDFS-3586
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 3.0.0
> Reporter: Brahma Reddy Battula
> Attachments: HDFS-3586-analysis.txt
>
>
> Scenario:
> =========
> Started four DN's(Say DN1,DN2,DN3 and DN4)
> writing files with RF=3..
> formed pipeline with DN1->DN2->DN3.
> Since DN3 network is very slow.it's not able to send acks.
> Again pipeline is fromed with DN1->DN2->DN4.
> Here DN4 network is also slow.
> So finally commitblocksync happend tp DN1 and DN2 successfully.
> block present in all the four DN's(finalized state in two DN's and rbw state in another DN's)..
> Here NN is asking replicate to DN3 and DN4,but it's failing since replcia's are already present in RBW dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3586) Blocks are not getting replicate
even DN's are availble.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13407410#comment-13407410 ]
Uma Maheswara Rao G commented on HDFS-3586:
-------------------------------------------
Thanks Konstantin.
Assigning it to Amith as he started working on this change.
> Blocks are not getting replicate even DN's are availble.
> --------------------------------------------------------
>
> Key: HDFS-3586
> URL: https://issues.apache.org/jira/browse/HDFS-3586
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 3.0.0
> Reporter: Brahma Reddy Battula
> Attachments: HDFS-3586-analysis.txt
>
>
> Scenario:
> =========
> Started four DN's(Say DN1,DN2,DN3 and DN4)
> writing files with RF=3..
> formed pipeline with DN1->DN2->DN3.
> Since DN3 network is very slow.it's not able to send acks.
> Again pipeline is fromed with DN1->DN2->DN4.
> Here DN4 network is also slow.
> So finally commitblocksync happend tp DN1 and DN2 successfully.
> block present in all the four DN's(finalized state in two DN's and rbw state in another DN's)..
> Here NN is asking replicate to DN3 and DN4,but it's failing since replcia's are already present in RBW dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3586) Blocks are not getting replicate
even DN's are availble.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405484#comment-13405484 ]
Uma Maheswara Rao G commented on HDFS-3586:
-------------------------------------------
Hi Konstantin,
{code}
I don't see how your suggestion solves the problem, as it doesn't work in case when all replicas are corrupt, if I understand it correctly.
{code}
When all replicas corrupt, we need not expect it to be replicate, because there is no good replica to replicate.
But in this case we have 2 good replicas. What I am proposing is, keeping minimum of 3 replicas(replication) is fine((good + corrupt) (or) (all 3 corrupt) (or) (all 3 good replicas)), more than that if we get any replica corrupt, let's invalidate it.SO, that even if we have single good replica, it will be copied to other node in this kind of situation. Ideally having whole cluster filled with corrupt replica and good replicas is a very rare case and almost no possibility in bigger clusters. Worth considering for smaller clusters also. Because we have cluster size more than replication size here, user expectation is replica will be replicated properly.
> Blocks are not getting replicate even DN's are availble.
> --------------------------------------------------------
>
> Key: HDFS-3586
> URL: https://issues.apache.org/jira/browse/HDFS-3586
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 3.0.0
> Reporter: Brahma Reddy Battula
> Attachments: HDFS-3586-analysis.txt
>
>
> Scenario:
> =========
> Started four DN's(Say DN1,DN2,DN3 and DN4)
> writing files with RF=3..
> formed pipeline with DN1->DN2->DN3.
> Since DN3 network is very slow.it's not able to send acks.
> Again pipeline is fromed with DN1->DN2->DN4.
> Here DN4 network is also slow.
> So finally commitblocksync happend tp DN1 and DN2 successfully.
> block present in all the four DN's(finalized state in two DN's and rbw state in another DN's)..
> Here NN is asking replicate to DN3 and DN4,but it's failing since replcia's are already present in RBW dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3586) Blocks are not getting replicate
even DN's are availble.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405293#comment-13405293 ]
Uma Maheswara Rao G commented on HDFS-3586:
-------------------------------------------
Hi Konstantin,
What is your suggestion on this issue?
Do you have any alternative idea in your mind?
Thanks
Uma
> Blocks are not getting replicate even DN's are availble.
> --------------------------------------------------------
>
> Key: HDFS-3586
> URL: https://issues.apache.org/jira/browse/HDFS-3586
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 3.0.0
> Reporter: Brahma Reddy Battula
> Attachments: HDFS-3586-analysis.txt
>
>
> Scenario:
> =========
> Started four DN's(Say DN1,DN2,DN3 and DN4)
> writing files with RF=3..
> formed pipeline with DN1->DN2->DN3.
> Since DN3 network is very slow.it's not able to send acks.
> Again pipeline is fromed with DN1->DN2->DN4.
> Here DN4 network is also slow.
> So finally commitblocksync happend tp DN1 and DN2 successfully.
> block present in all the four DN's(finalized state in two DN's and rbw state in another DN's)..
> Here NN is asking replicate to DN3 and DN4,but it's failing since replcia's are already present in RBW dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3586) Blocks are not getting replicate even
DN's are availble.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G reassigned HDFS-3586:
-----------------------------------------
Assignee: amith
> Blocks are not getting replicate even DN's are availble.
> --------------------------------------------------------
>
> Key: HDFS-3586
> URL: https://issues.apache.org/jira/browse/HDFS-3586
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 3.0.0
> Reporter: Brahma Reddy Battula
> Assignee: amith
> Attachments: HDFS-3586-analysis.txt
>
>
> Scenario:
> =========
> Started four DN's(Say DN1,DN2,DN3 and DN4)
> writing files with RF=3..
> formed pipeline with DN1->DN2->DN3.
> Since DN3 network is very slow.it's not able to send acks.
> Again pipeline is fromed with DN1->DN2->DN4.
> Here DN4 network is also slow.
> So finally commitblocksync happend tp DN1 and DN2 successfully.
> block present in all the four DN's(finalized state in two DN's and rbw state in another DN's)..
> Here NN is asking replicate to DN3 and DN4,but it's failing since replcia's are already present in RBW dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3586) Blocks are not getting replicate
even DN's are availble.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431862#comment-13431862 ]
Uma Maheswara Rao G commented on HDFS-3586:
-------------------------------------------
{quote}
even (countNodes(b.stored).liveReplicas() + countNodes(b.stored).corruptReplicas()) < bc.getReplication()).
{quote}
This condition may delete the blocks when all replicas corrupted for a file. I think, our replication design don't want to delete when all replicas corrupted as anyway you can't replicate them anymore.
how about the below condition?
{quote}
if ((countNodes(b.stored).liveReplicas() + countNodes(b.stored).corruptReplicas()) >= bc.getReplication() && countNodes(b.stored).liveReplicas() > minReplication)
{quote}
> Blocks are not getting replicate even DN's are availble.
> --------------------------------------------------------
>
> Key: HDFS-3586
> URL: https://issues.apache.org/jira/browse/HDFS-3586
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 2.0.0-alpha, 2.1.0-alpha, 3.0.0
> Reporter: Brahma Reddy Battula
> Assignee: amith
> Attachments: HDFS-3586-analysis.txt
>
>
> Scenario:
> =========
> Started four DN's(Say DN1,DN2,DN3 and DN4)
> writing files with RF=3..
> formed pipeline with DN1->DN2->DN3.
> Since DN3 network is very slow.it's not able to send acks.
> Again pipeline is fromed with DN1->DN2->DN4.
> Here DN4 network is also slow.
> So finally commitblocksync happend tp DN1 and DN2 successfully.
> block present in all the four DN's(finalized state in two DN's and rbw state in another DN's)..
> Here NN is asking replicate to DN3 and DN4,but it's failing since replcia's are already present in RBW dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3586) Blocks are not getting replicate
even DN's are availble.
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405178#comment-13405178 ]
Uma Maheswara Rao G commented on HDFS-3586:
-------------------------------------------
Thanks Brahma for filing the JIRA.
This is similar to HDFS-3493. But here the change is, DNs are available more than replication. So, ideally block should get replicated.
The problem here is, you have 2 live replicas and in remaining 2 DNs you have partial block present in RBW. So, when NN tries to replicate, DN will reject them saying block already exist in RBW. So, your replication may not happen even though you have more nodes.
Here I think the possible fix could be that, we should change the below condition
*if (countNodes(b.stored).liveReplicas() >= bc.getReplication()) {*
to something like *if ((countNodes(b.stored).liveReplicas() + countNodes(b.stored).corruptReplicas()) > bc.getReplication()) {*
So, the extra corrupted blocks(more than replication) will get invalidated and later replication can work normally.
> Blocks are not getting replicate even DN's are availble.
> --------------------------------------------------------
>
> Key: HDFS-3586
> URL: https://issues.apache.org/jira/browse/HDFS-3586
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 3.0.0
> Reporter: Brahma Reddy Battula
> Attachments: HDFS-3586-analysis.txt
>
>
> Scenario:
> =========
> Started four DN's(Say DN1,DN2,DN3 and DN4)
> writing files with RF=3..
> formed pipeline with DN1->DN2->DN3.
> Since DN3 network is very slow.it's not able to send acks.
> Again pipeline is fromed with DN1->DN2->DN4.
> Here DN4 network is also slow.
> So finally commitblocksync happend tp DN1 and DN2 successfully.
> block present in all the four DN's(finalized state in two DN's and rbw state in another DN's)..
> Here NN is asking replicate to DN3 and DN4,but it's failing since replcia's are already present in RBW dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3586) Blocks are not getting replicate
even DN's are availble.
Posted by "Han Xiao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431831#comment-13431831 ]
Han Xiao commented on HDFS-3586:
--------------------------------
Hi Uma, why not provide a configuration item. Then user can decide whether to delete the corrupt replica as long as good one exists()(even (countNodes(b.stored).liveReplicas() + countNodes(b.stored).corruptReplicas()) < bc.getReplication()). This will be very helpful for small cluster.
> Blocks are not getting replicate even DN's are availble.
> --------------------------------------------------------
>
> Key: HDFS-3586
> URL: https://issues.apache.org/jira/browse/HDFS-3586
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 2.0.0-alpha, 2.1.0-alpha, 3.0.0
> Reporter: Brahma Reddy Battula
> Assignee: amith
> Attachments: HDFS-3586-analysis.txt
>
>
> Scenario:
> =========
> Started four DN's(Say DN1,DN2,DN3 and DN4)
> writing files with RF=3..
> formed pipeline with DN1->DN2->DN3.
> Since DN3 network is very slow.it's not able to send acks.
> Again pipeline is fromed with DN1->DN2->DN4.
> Here DN4 network is also slow.
> So finally commitblocksync happend tp DN1 and DN2 successfully.
> block present in all the four DN's(finalized state in two DN's and rbw state in another DN's)..
> Here NN is asking replicate to DN3 and DN4,but it's failing since replcia's are already present in RBW dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira