You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (Created) (JIRA)" <ji...@apache.org> on 2011/10/18 04:38:11 UTC
[jira] [Created] (HDFS-2465) Add HDFS support for fadvise readahead
and drop-behind
Add HDFS support for fadvise readahead and drop-behind
------------------------------------------------------
Key: HDFS-2465
URL: https://issues.apache.org/jira/browse/HDFS-2465
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137683#comment-13137683 ]
Todd Lipcon commented on HDFS-2465:
-----------------------------------
looks like the test-patch build ran before HADOOP-7753 was integrated in the snapshot build. I'll wait half an hour or so and retrigger this test-patch. Meanwhile, should be good to review.
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2465) Add HDFS support for fadvise readahead
and drop-behind
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-2465:
----------------------------------
Fix Version/s: (was: 1.2.0)
1.1.0
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0, 1.1.0
>
> Attachments: HDFS-2465.branch-1.patch, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138961#comment-13138961 ]
Hudson commented on HDFS-2465:
------------------------------
Integrated in Hadoop-Common-0.23-Commit #100 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/100/])
HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138840#comment-13138840 ]
Eli Collins commented on HDFS-2465:
-----------------------------------
+1 latest patch lgtm
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139228#comment-13139228 ]
Hudson commented on HDFS-2465:
------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #54 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/54/])
HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2465) Add HDFS support for fadvise readahead
and drop-behind
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2465:
------------------------------
Component/s: performance
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138884#comment-13138884 ]
Hudson commented on HDFS-2465:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1216 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1216/])
HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2465) Add HDFS support for fadvise readahead
and drop-behind
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-2465:
----------------------------------
Fix Version/s: 1.2.0
I committed the patch to branch-1
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0, 1.2.0
>
> Attachments: HDFS-2465.branch-1.patch, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2465) Add HDFS support for fadvise readahead
and drop-behind
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2465:
------------------------------
Attachment: hdfs-2465.txt
reuploading patch to trigger build. Also fixed the style nit Eli mentioned above.
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138745#comment-13138745 ]
Todd Lipcon commented on HDFS-2465:
-----------------------------------
Failing tests seem unrelated. I ran the webhdfs tests locally and they passed. TestHDFSCLI fails on my local box, but does so on trunk as well for me.
bq. -1 tests included. The patch doesn't appear to include any new or modified tests.
Unfortunately it's not really possible to unit test this. I ran through several terasorts on a small test cluster with this patch and verified that it helps performance and didn't cause stability issues. I've been testing with very similar patches for a couple weeks and seen no issues. It also is off-by-default.
Given Eli gave a +1 earlier, I will wait a few hours and commit this if there are no further comments.
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139257#comment-13139257 ]
Hudson commented on HDFS-2465:
------------------------------
Integrated in Hadoop-Hdfs-trunk #847 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/847/])
HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138722#comment-13138722 ]
Hadoop QA commented on HDFS-2465:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501334/hdfs-2465.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.cli.TestHDFSCLI
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1487//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1487//console
This message is automatically generated.
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137926#comment-13137926 ]
Hadoop QA commented on HDFS-2465:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501194/hdfs-2465.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
org.apache.hadoop.hdfs.TestSmallBlock
org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
org.apache.hadoop.hdfs.server.balancer.TestBalancer
org.apache.hadoop.hdfs.TestFileCreation
org.apache.hadoop.hdfs.TestFileCreationClient
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
org.apache.hadoop.hdfs.TestDFSClientExcludedNodes
org.apache.hadoop.hdfs.TestSetrepIncreasing
org.apache.hadoop.hdfs.TestPread
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1480//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1480//console
This message is automatically generated.
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2465) Add HDFS support for fadvise readahead
and drop-behind
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2465:
------------------------------
Status: Patch Available (was: Open)
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412361#comment-13412361 ]
Brandon Li commented on HDFS-2465:
----------------------------------
Thanks. I ran teragen/terasort/teravalidate/TestDFSIO tests.
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: HDFS-2465.branch-1.patch, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2465) Add HDFS support for fadvise readahead
and drop-behind
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2465:
------------------------------
Attachment: hdfs-2465.txt
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137718#comment-13137718 ]
Eli Collins commented on HDFS-2465:
-----------------------------------
bq. I don't think so. HDFS doesn't really do readahead except for filling up the TCP window
I was thinking that filling up the dfs packet is essentially readahead since we'll read the whole 64kb (or whatever is configured) regardless of how much the client asks for but yea in this context that's not relevant (eg HBase won't issue such small reads).
+1 to your latest patch. (Nit: add spaces in "1024*1024" no need to generate a new patch)
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138930#comment-13138930 ]
Hudson commented on HDFS-2465:
------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #100 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/100/])
HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139279#comment-13139279 ]
Hudson commented on HDFS-2465:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #878 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/878/])
HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137673#comment-13137673 ]
Hadoop QA commented on HDFS-2465:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501177/hdfs-2465.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed the unit tests build
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1477//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1477//console
This message is automatically generated.
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138941#comment-13138941 ]
Hudson commented on HDFS-2465:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1269 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1269/])
HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2465) Add HDFS support for fadvise readahead
and drop-behind
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2465:
------------------------------
Attachment: hdfs-2465.txt
Had forgotten to deal with the simulated FSDataset in the previous patch. New patch checks whether the block stream is a FileInputStream before attempting to get its file descriptor.
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412320#comment-13412320 ]
Suresh Srinivas commented on HDFS-2465:
---------------------------------------
+! for the branch-1 patch. Please comment on the tests you have run.
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: HDFS-2465.branch-1.patch, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413190#comment-13413190 ]
Suresh Srinivas commented on HDFS-2465:
---------------------------------------
This time, going from +! to +1 :-)
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: HDFS-2465.branch-1.patch, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2465) Add HDFS support for fadvise readahead
and drop-behind
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2465:
------------------------------
Resolution: Fixed
Fix Version/s: 0.23.0
Release Note:
HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache. This support is currently considered experimental, and may be enabled by configuring the following keys:
dfs.datanode.drop.cache.behind.writes - set to true to drop data out of the buffer cache after writing
dfs.datanode.drop.cache.behind.reads - set to true to drop data out of the buffer cache when performing sequential reads
dfs.datanode.sync.behind.writes - set to true to trigger dirty page writeback immediately after writing data
dfs.datanode.readahead.bytes - set to a non-zero value to trigger readahead for sequential reads
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk and 23. Thanks for the review, Eli.
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139302#comment-13139302 ]
Hudson commented on HDFS-2465:
------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #72 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/72/])
HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2465) Add HDFS support for fadvise readahead
and drop-behind
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Li updated HDFS-2465:
-----------------------------
Attachment: HDFS-2465.branch-1.patch
Back port it to branch-1.
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: HDFS-2465.branch-1.patch, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137667#comment-13137667 ]
Todd Lipcon commented on HDFS-2465:
-----------------------------------
bq. There isn't any config that enables syncBehindWrites in BlockerRecevier
Woops! good catch. Lost that somewhere along the line.
bq. Would it be more clear to express LONG_READ_THRESHOLD in terms of a multiple of the packet size or the IO file buffer size? Seems like the host readahead should match the implicit HDFS readahead.
I don't think so. HDFS doesn't really do readahead except for filling up the TCP window. But submitting very large (multi-MB)readahead to the OS lets it schedule big sequential reads. So, I don't think there's any real correlation. To set this tunable, I'd probably do some mental math like: "well, I probably have 20 files being read at once, so 8M of readahead on each of those will end up being 160M of RAM used." Having to multiply by other factors is harder to follow.
Fixed the other nits. New patch momentarily.
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2465) Add HDFS support for fadvise readahead
and drop-behind
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2465:
------------------------------
Attachment: hdfs-2465.txt
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137374#comment-13137374 ]
Eli Collins commented on HDFS-2465:
-----------------------------------
Looked at the patch, think this is a reasonable first DN-wide heuristic. Comments follow:
* There isn't any config that enables syncBehindWrites in BlockerRecevier.
* Would it be more clear to express LONG_READ_THRESHOLD in terms of a multiple of the packet size or the IO file buffer size? Seems like the host readahead should match the implicit HDFS readahead.
* In BlockerSender#sendBlock by the 1st fadvise a comment like "advice applies to the entire file" would help
* Nit: add _BYTES to LONG_READ_THRESHOLD and CACHE_DROP_INTERVAL.
Otherwise looks great.
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138950#comment-13138950 ]
Hudson commented on HDFS-2465:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #1193 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1193/])
HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise
readahead and drop-behind
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138913#comment-13138913 ]
Hudson commented on HDFS-2465:
------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #105 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/105/])
HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>
> Key: HDFS-2465
> URL: https://issues.apache.org/jira/browse/HDFS-2465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt
>
>
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira