You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2012/07/20 21:11:35 UTC
[jira] [Created] (HDFS-3697) Enable fadvise readahead by default
Todd Lipcon created HDFS-3697:
---------------------------------
Summary: Enable fadvise readahead by default
Key: HDFS-3697
URL: https://issues.apache.org/jira/browse/HDFS-3697
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node, performance
Affects Versions: 3.0.0, 2.2.0-alpha
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Minor
The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable them by default in future versions so that users get this benefit without any manual configuration required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427476#comment-13427476 ]
Eli Collins commented on HDFS-3697:
-----------------------------------
Agree with Brandon that this seems reasonable for branch-1, I'll merge the patch.
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421430#comment-13421430 ]
Hudson commented on HDFS-3697:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1146 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1146/])
HDFS-3697. Enable fadvise readahead by default. Contributed by Todd Lipcon. (Revision 1364698)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364698
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HDFS-3697) Enable fadvise readahead by default
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed HDFS-3697.
-------------------------------
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 1.2.0, 3.0.0, 2.0.2-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3697) Enable fadvise readahead by default
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3697:
------------------------------
Description:
The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
was:The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable them by default in future versions so that users get this benefit without any manual configuration required.
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419474#comment-13419474 ]
Hadoop QA commented on HDFS-3697:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537392/hdfs-3697.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javac. The patch appears to cause the build to fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2879//console
This message is automatically generated.
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable them by default in future versions so that users get this benefit without any manual configuration required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420793#comment-13420793 ]
Todd Lipcon commented on HDFS-3697:
-----------------------------------
You mean the datanode, right?
It would be worth considering adding these as per-stream flags - e.g an HDFS "fadvise"-like API on DFSInputStream/DFSOutputStream that changes the behavior. Then, the datanode-wide flags would just serve as defaults.
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422767#comment-13422767 ]
Suresh Srinivas commented on HDFS-3697:
---------------------------------------
Todd, I think we should add appropriate release notes, even though this is no incompatible, to call out attention to enabling this feature by default.
BTW this is a good change!
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419664#comment-13419664 ]
Hadoop QA commented on HDFS-3697:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537405/hdfs-3697.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2880//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2880//console
This message is automatically generated.
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427485#comment-13427485 ]
Eli Collins commented on HDFS-3697:
-----------------------------------
+1 to Todd's branch-1 patch.
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420768#comment-13420768 ]
Hadoop QA commented on HDFS-3697:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537579/hdfs-3697-branch-1.txt
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2887//console
This message is automatically generated.
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421051#comment-13421051 ]
Todd Lipcon commented on HDFS-3697:
-----------------------------------
Hey Brandon. In the absence of the improvement discussed above, do you think we should commit to branch-1? I ask, given you've been doing the other fadvise backports into branch-1.
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3697) Enable fadvise readahead by default
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3697:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've merged this.
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 1.2.0, 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420764#comment-13420764 ]
Hudson commented on HDFS-3697:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #2511 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2511/])
HDFS-3697. Enable fadvise readahead by default. Contributed by Todd Lipcon. (Revision 1364698)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364698
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3697) Enable fadvise readahead by default
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3697:
------------------------------
Release Note: The datanode now performs 4MB readahead by default when reading data from its disks, if the native libraries are present. This has been shown to improve performance in many workloads. The feature may be disabled by setting dfs.datanode.readahead.bytes to "0".
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3697) Enable fadvise readahead by default
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3697:
------------------------------
Attachment: hdfs-3697.txt
Oops, typo in the XML made it malformed. New one passes xmllint.
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable them by default in future versions so that users get this benefit without any manual configuration required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3697) Enable fadvise readahead by default
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3697:
------------------------------
Fix Version/s: 2.2.0-alpha
3.0.0
Hadoop Flags: Reviewed
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419539#comment-13419539 ]
Eli Collins commented on HDFS-3697:
-----------------------------------
+1 pending jenkins. It's worth calling out in the description that this only enables readahead, not drop cache and sync.
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable them by default in future versions so that users get this benefit without any manual configuration required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3697) Enable fadvise readahead by default
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3697:
------------------------------
Status: Patch Available (was: Open)
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable them by default in future versions so that users get this benefit without any manual configuration required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420799#comment-13420799 ]
Hudson commented on HDFS-3697:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2532 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2532/])
HDFS-3697. Enable fadvise readahead by default. Contributed by Todd Lipcon. (Revision 1364698)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364698
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420763#comment-13420763 ]
Hudson commented on HDFS-3697:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2576 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2576/])
HDFS-3697. Enable fadvise readahead by default. Contributed by Todd Lipcon. (Revision 1364698)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364698
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421075#comment-13421075 ]
Brandon Li commented on HDFS-3697:
----------------------------------
Hey Todd, I've seen some performance improvement in my branch-1 test with the fadvise support enabled. I think should commit it to branch-1.
The thing I am not sure is the readahead step size since I didn't try out with enough workloads and step sizes in my tests. Given 4MB is good in your experiments and it's also configurable, we can start with it.
I just feel it's not trivial sometimes to guess out a good step size beforehand. :-) If it's possible in the future, we might want to make readahead adjust its step size based on observed access pattern(however, tracking multiple parallel streams' access pattern can be same challenging :-( ).
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420796#comment-13420796 ]
Brandon Li commented on HDFS-3697:
----------------------------------
[quote]You mean the datanode, right?[quote]
ops, yes.
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420792#comment-13420792 ]
Brandon Li commented on HDFS-3697:
----------------------------------
Do we have any good way to enable/disable readahead, drop cache and sync without restarting namenode?
Just thinking when the workload changes, the customer has to restart the namenode to take advantage of the performance enhancement.
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3697) Enable fadvise readahead by default
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3697:
------------------------------
Target Version/s: (was: 1.2.0, 3.0.0, 2.2.0-alpha)
Fix Version/s: 1.2.0
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 1.2.0, 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3697) Enable fadvise readahead by default
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421374#comment-13421374 ]
Hudson commented on HDFS-3697:
------------------------------
Integrated in Hadoop-Hdfs-trunk #1114 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1114/])
HDFS-3697. Enable fadvise readahead by default. Contributed by Todd Lipcon. (Revision 1364698)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364698
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3697) Enable fadvise readahead by default
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3697:
------------------------------
Attachment: hdfs-3697.txt
Attached patch enables the readahead to 4MB by default. Experimentally we've determined this provides a good performance boost without too high an increase in buffer cache usage.
I also took the liberty of adding documentation for the other fadvise parameters to hdfs-default.xml in this patch.
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable them by default in future versions so that users get this benefit without any manual configuration required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3697) Enable fadvise readahead by default
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3697:
------------------------------
Attachment: hdfs-3697-branch-1.txt
Committed to trunk and branch-2. Here is a branch-1 patch. What do folks think about backporting?
> Enable fadvise readahead by default
> -----------------------------------
>
> Key: HDFS-3697
> URL: https://issues.apache.org/jira/browse/HDFS-3697
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: hdfs-3697-branch-1.txt, hdfs-3697.txt, hdfs-3697.txt
>
>
> The fadvise features have been implemented for some time, and we've enabled them in production at a lot of customer sites without difficulty. I'd like to enable the readahead feature by default in future versions so that users get this benefit without any manual configuration required.
> The other fadvise features seem to be more workload-dependent and need further testing before enabling by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira