You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Pete Wyckoff (JIRA)" <ji...@apache.org> on 2008/07/22 04:38:31 UTC
[jira] Created: (HADOOP-3805) improve fuse-dfs write performance
which is 33% slower than hadoop dfs -copyFromLocal
improve fuse-dfs write performance which is 33% slower than hadoop dfs -copyFromLocal
-------------------------------------------------------------------------------------
Key: HADOOP-3805
URL: https://issues.apache.org/jira/browse/HADOOP-3805
Project: Hadoop Core
Issue Type: Improvement
Components: contrib/fuse-dfs
Reporter: Pete Wyckoff
Priority: Minor
on hadoop17 running on the namenode/fuse mount point, fuse is 33% slower for writing a 1GB file. I don't know right now how to improve this but thought I should open up a JIRA for it.
Note - this benchmark is with no tuning or special fuse config params.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3805) improve fuse-dfs write performance
which is 33% slower than hadoop dfs -copyFromLocal
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620094#action_12620094 ]
Hadoop QA commented on HADOOP-3805:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12387356/hadoop-3805.txt
against trunk revision 682978.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3004/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3004/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3004/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3004/console
This message is automatically generated.
> improve fuse-dfs write performance which is 33% slower than hadoop dfs -copyFromLocal
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-3805
> URL: https://issues.apache.org/jira/browse/HADOOP-3805
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: hadoop-3805.txt
>
>
> on hadoop17 running on the namenode/fuse mount point, fuse is 33% slower for writing a 1GB file. I don't know right now how to improve this but thought I should open up a JIRA for it.
> Note - this benchmark is with no tuning or special fuse config params.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3805) improve fuse-dfs write performance
which is 33% slower than hadoop dfs -copyFromLocal
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624668#action_12624668 ]
Hudson commented on HADOOP-3805:
--------------------------------
Integrated in Hadoop-trunk #581 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/])
> improve fuse-dfs write performance which is 33% slower than hadoop dfs -copyFromLocal
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-3805
> URL: https://issues.apache.org/jira/browse/HADOOP-3805
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: hadoop-3805.txt
>
>
> on hadoop17 running on the namenode/fuse mount point, fuse is 33% slower for writing a 1GB file. I don't know right now how to improve this but thought I should open up a JIRA for it.
> Note - this benchmark is with no tuning or special fuse config params.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3805) improve fuse-dfs write performance
which is 33% slower than hadoop dfs -copyFromLocal
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12619009#action_12619009 ]
Pete Wyckoff commented on HADOOP-3805:
--------------------------------------
Discussion of options and kernels needed for better write performance in fuse:
http://www.nabble.com/fuse-performance-td18271595.html
basically need to use -o big_writes and have a kernel > 2.6.26-rc*
I don't currently have access to such a kernel, but it's probably worth while just adding the -obig_writes option by default and having a conifg vars no_big_writes use_big_writes.
I will do up a patch.
> improve fuse-dfs write performance which is 33% slower than hadoop dfs -copyFromLocal
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-3805
> URL: https://issues.apache.org/jira/browse/HADOOP-3805
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
>
> on hadoop17 running on the namenode/fuse mount point, fuse is 33% slower for writing a 1GB file. I don't know right now how to improve this but thought I should open up a JIRA for it.
> Note - this benchmark is with no tuning or special fuse config params.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3805) improve fuse-dfs write performance
which is 33% slower than hadoop dfs -copyFromLocal
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3805:
---------------------------------
Status: Patch Available (was: Open)
> improve fuse-dfs write performance which is 33% slower than hadoop dfs -copyFromLocal
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-3805
> URL: https://issues.apache.org/jira/browse/HADOOP-3805
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: hadoop-3805.txt
>
>
> on hadoop17 running on the namenode/fuse mount point, fuse is 33% slower for writing a 1GB file. I don't know right now how to improve this but thought I should open up a JIRA for it.
> Note - this benchmark is with no tuning or special fuse config params.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3805) improve fuse-dfs write performance
which is 33% slower than hadoop dfs -copyFromLocal
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3805:
---------------------------------
Attachment: hadoop-3805.txt
I made -obig_writes opt in so it wouldn't cause a backwards compat problem. I also #ifdef the actual use of the flag because it's only supported in fuse 2.8.0 + and 2.7.4 is the current stable release.
> improve fuse-dfs write performance which is 33% slower than hadoop dfs -copyFromLocal
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-3805
> URL: https://issues.apache.org/jira/browse/HADOOP-3805
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: hadoop-3805.txt
>
>
> on hadoop17 running on the namenode/fuse mount point, fuse is 33% slower for writing a 1GB file. I don't know right now how to improve this but thought I should open up a JIRA for it.
> Note - this benchmark is with no tuning or special fuse config params.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3805) improve fuse-dfs write performance
which is 33% slower than hadoop dfs -copyFromLocal
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-3805:
-------------------------------
Resolution: Fixed
Fix Version/s: 0.19.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed.
> improve fuse-dfs write performance which is 33% slower than hadoop dfs -copyFromLocal
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-3805
> URL: https://issues.apache.org/jira/browse/HADOOP-3805
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: hadoop-3805.txt
>
>
> on hadoop17 running on the namenode/fuse mount point, fuse is 33% slower for writing a 1GB file. I don't know right now how to improve this but thought I should open up a JIRA for it.
> Note - this benchmark is with no tuning or special fuse config params.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3805) improve fuse-dfs write performance
which is 33% slower than hadoop dfs -copyFromLocal
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12619010#action_12619010 ]
Pete Wyckoff commented on HADOOP-3805:
--------------------------------------
I should also mention that hdfs_write utility build on libhdfs is no slower than the shell copy, so the problem is likely not in libhdfs. (based on very simple tests and Owen also confirmed that he thought libhdfs wouldn't have any performance issues)
> improve fuse-dfs write performance which is 33% slower than hadoop dfs -copyFromLocal
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-3805
> URL: https://issues.apache.org/jira/browse/HADOOP-3805
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
>
> on hadoop17 running on the namenode/fuse mount point, fuse is 33% slower for writing a 1GB file. I don't know right now how to improve this but thought I should open up a JIRA for it.
> Note - this benchmark is with no tuning or special fuse config params.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3805) improve fuse-dfs write performance
which is 33% slower than hadoop dfs -copyFromLocal
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620324#action_12620324 ]
dhruba borthakur commented on HADOOP-3805:
------------------------------------------
+1 Code looks good.
> improve fuse-dfs write performance which is 33% slower than hadoop dfs -copyFromLocal
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-3805
> URL: https://issues.apache.org/jira/browse/HADOOP-3805
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: hadoop-3805.txt
>
>
> on hadoop17 running on the namenode/fuse mount point, fuse is 33% slower for writing a 1GB file. I don't know right now how to improve this but thought I should open up a JIRA for it.
> Note - this benchmark is with no tuning or special fuse config params.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.