You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Tomasz Nykiel <to...@fb.com> on 2011/07/01 22:32:12 UTC
Review Request: HDFS-395 DFS Scalability: Incremental block reports
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/998/
-----------------------------------------------------------
Review request for hadoop-hdfs, Dhruba Borthakur and Hairong Kuang.
Summary
-------
This patch introduces explicit ACKs sent form datanodes to the namenode in order to minimize the difference between NN blocks info and DN state.
This will allow for sending the full block reports less frequently, which in turn will minimize the overhead of processing them, and blocking the namenode for extended time.
This addresses bug HDFS-395.
https://issues.apache.org/jira/browse/HDFS-395
Diffs
-----
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java 1142055
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java 1142055
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java 1142055
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java 1142055
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java 1142055
trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java 1142055
trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java 1142055
Diff: https://reviews.apache.org/r/998/diff
Testing
-------
Will do JUnit tests.
Thanks,
Tomasz
Re: Review Request: HDFS-395 DFS Scalability: Incremental block reports
Posted by Tomasz Nykiel <tn...@cs.toronto.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/998/
-----------------------------------------------------------
(Updated 2011-08-24 03:33:32.850921)
Review request for hadoop-hdfs, Dhruba Borthakur and Hairong Kuang.
Summary
-------
This patch introduces explicit ACKs sent form datanodes to the namenode in order to minimize the difference between NN blocks info and DN state.
This will allow for sending the full block reports less frequently, which in turn will minimize the overhead of processing them, and blocking the namenode for extended time.
This addresses bug HDFS-395.
https://issues.apache.org/jira/browse/HDFS-395
Diffs (updated)
-----
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java 1160950
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java 1160950
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java 1160950
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java 1160950
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java 1160950
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java 1160950
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java 1160950
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/ReceivedDeletedBlockInfo.java PRE-CREATION
trunk/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java 1160950
trunk/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java 1160950
Diff: https://reviews.apache.org/r/998/diff
Testing
-------
Will do JUnit tests.
Thanks,
Tomasz
Re: Review Request: HDFS-395 DFS Scalability: Incremental block reports
Posted by Tomasz Nykiel <tn...@cs.toronto.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/998/
-----------------------------------------------------------
(Updated 2011-08-23 22:55:47.297820)
Review request for hadoop-hdfs, Dhruba Borthakur and Hairong Kuang.
Changes
-------
Adjusted to recent code changes.
Fized offerService comment.
Summary
-------
This patch introduces explicit ACKs sent form datanodes to the namenode in order to minimize the difference between NN blocks info and DN state.
This will allow for sending the full block reports less frequently, which in turn will minimize the overhead of processing them, and blocking the namenode for extended time.
This addresses bug HDFS-395.
https://issues.apache.org/jira/browse/HDFS-395
Diffs (updated)
-----
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java 1160919
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java 1160919
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java 1160919
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java 1160919
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java 1160919
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java 1160919
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java 1160919
trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/ReceivedDeletedBlockInfo.java PRE-CREATION
trunk/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java 1160919
trunk/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java 1160919
Diff: https://reviews.apache.org/r/998/diff
Testing
-------
Will do JUnit tests.
Thanks,
Tomasz
Re: Review Request: HDFS-395 DFS Scalability: Incremental block reports
Posted by Tomasz Nykiel <to...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/998/
-----------------------------------------------------------
(Updated 2011-07-28 00:40:14.000447)
Review request for hadoop-hdfs, Dhruba Borthakur and Hairong Kuang.
Changes
-------
Addressed the comments.
Summary
-------
This patch introduces explicit ACKs sent form datanodes to the namenode in order to minimize the difference between NN blocks info and DN state.
This will allow for sending the full block reports less frequently, which in turn will minimize the overhead of processing them, and blocking the namenode for extended time.
This addresses bug HDFS-395.
https://issues.apache.org/jira/browse/HDFS-395
Diffs (updated)
-----
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java 1151690
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java 1151690
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java 1151690
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java 1151690
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java 1151690
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java 1151690
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java 1151690
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java 1151690
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/ReceivedDeletedBlockInfo.java PRE-CREATION
trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java 1151690
trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java 1151690
Diff: https://reviews.apache.org/r/998/diff
Testing
-------
Will do JUnit tests.
Thanks,
Tomasz
Re: Review Request: HDFS-395 DFS Scalability: Incremental block reports
Posted by Tomasz Nykiel <to...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/998/
-----------------------------------------------------------
(Updated 2011-07-11 21:06:02.473171)
Review request for hadoop-hdfs, Dhruba Borthakur and Hairong Kuang.
Changes
-------
- I removed renaming of the block and meta file from FSDataset. We should consider it in a separate diff, as suggested by Hairong. By doing so, I need to defer the notification until the FSAsyncDiskService deletes the files.
- I added an optimization, which does not procuce ACKs for blocks which come from entirely deleted files (in this situation, the blocks are instantly removed from the blocks map at the NN side, and hence we need no ACK).
Summary
-------
This patch introduces explicit ACKs sent form datanodes to the namenode in order to minimize the difference between NN blocks info and DN state.
This will allow for sending the full block reports less frequently, which in turn will minimize the overhead of processing them, and blocking the namenode for extended time.
This addresses bug HDFS-395.
https://issues.apache.org/jira/browse/HDFS-395
Diffs (updated)
-----
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java 1145346
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java 1145346
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java 1145346
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java 1145346
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java 1145346
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java 1145346
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java 1145346
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java 1145346
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/ReceivedDeletedBlockInfo.java PRE-CREATION
trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java 1145346
trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java 1145346
Diff: https://reviews.apache.org/r/998/diff
Testing
-------
Will do JUnit tests.
Thanks,
Tomasz
Re: Review Request: HDFS-395 DFS Scalability: Incremental block reports
Posted by Tomasz Nykiel <to...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/998/
-----------------------------------------------------------
(Updated 2011-07-01 20:34:17.116645)
Review request for hadoop-hdfs, Dhruba Borthakur and Hairong Kuang.
Summary
-------
This patch introduces explicit ACKs sent form datanodes to the namenode in order to minimize the difference between NN blocks info and DN state.
This will allow for sending the full block reports less frequently, which in turn will minimize the overhead of processing them, and blocking the namenode for extended time.
This addresses bug HDFS-395.
https://issues.apache.org/jira/browse/HDFS-395
Diffs (updated)
-----
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java 1142085
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java 1142085
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java 1142085
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java 1142085
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java 1142085
trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/ReceivedDeletedBlockInfo.java PRE-CREATION
trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java 1142085
trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java 1142085
Diff: https://reviews.apache.org/r/998/diff
Testing
-------
Will do JUnit tests.
Thanks,
Tomasz