You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2012/06/13 00:30:43 UTC
[jira] [Created] (HDFS-3529) Use direct buffers for data in write
path
Todd Lipcon created HDFS-3529:
---------------------------------
Summary: Use direct buffers for data in write path
Key: HDFS-3529
URL: https://issues.apache.org/jira/browse/HDFS-3529
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node, performance
Affects Versions: 2.0.0-alpha
Reporter: Todd Lipcon
Assignee: Todd Lipcon
The write path currently makes several unnecessary data copies in order to go to and from byte arrays. We can improve performance by using direct byte buffers to avoid the copy. This is also a prerequisite for native checksum calculation (HDFS-3528)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write
path
Posted by "Trevor Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485869#comment-13485869 ]
Trevor Robinson commented on HDFS-3529:
---------------------------------------
The two Jenkins failures seem to be the same issue:
# One {{BlockReceiver$PacketResponder}} thread is interrupted in flushing an ack to an upstream datanode.
# Another {{BlockReceiver$PacketResponder}} thread gets an {{EOFException}} trying to read the length prefix of an ack.
# An NPE occurs during {{PacketReceiver.mirrorPacketTo}} (called via {{DataXceiver.writeBlock}}) because the packet buffer is null, which should only happen if {{close}} has been called.
I'm not sure which of the three caused the others, or if there's some other cause. I ran hundreds of iterations of {{TestLoadGenerator}} in a loop, and cannot reproduce a failure locally. I'd appreciate any suggestions on how to go about reproducing it. For example, is it reasonable to attach a patch with extra logging, or does that disrupt builds for other issues? Or is it possible to get access to a machine with the same configuration?
> Use direct buffers for data in write path
> -----------------------------------------
>
> Key: HDFS-3529
> URL: https://issues.apache.org/jira/browse/HDFS-3529
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 2.0.0-alpha
> Reporter: Todd Lipcon
> Assignee: Trevor Robinson
> Attachments: HDFS-3529.patch
>
>
> The write path currently makes several unnecessary data copies in order to go to and from byte arrays. We can improve performance by using direct byte buffers to avoid the copy. This is also a prerequisite for native checksum calculation (HDFS-3528)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3529) Use direct buffers for data in write
path
Posted by "Trevor Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Trevor Robinson updated HDFS-3529:
----------------------------------
Status: Patch Available (was: Open)
> Use direct buffers for data in write path
> -----------------------------------------
>
> Key: HDFS-3529
> URL: https://issues.apache.org/jira/browse/HDFS-3529
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 2.0.0-alpha
> Reporter: Todd Lipcon
> Assignee: Trevor Robinson
> Attachments: HDFS-3529.patch
>
>
> The write path currently makes several unnecessary data copies in order to go to and from byte arrays. We can improve performance by using direct byte buffers to avoid the copy. This is also a prerequisite for native checksum calculation (HDFS-3528)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write
path
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491671#comment-13491671 ]
Hadoop QA commented on HDFS-3529:
---------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12552318/dfsio-x86-trunk-vs-3529.png
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3455//console
This message is automatically generated.
> Use direct buffers for data in write path
> -----------------------------------------
>
> Key: HDFS-3529
> URL: https://issues.apache.org/jira/browse/HDFS-3529
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 2.0.0-alpha
> Reporter: Todd Lipcon
> Assignee: Trevor Robinson
> Attachments: dfsio-x86-trunk-vs-3529.png, HDFS-3529.patch
>
>
> The write path currently makes several unnecessary data copies in order to go to and from byte arrays. We can improve performance by using direct byte buffers to avoid the copy. This is also a prerequisite for native checksum calculation (HDFS-3528)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write
path
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485777#comment-13485777 ]
Hadoop QA commented on HDFS-3529:
---------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12551137/HDFS-3529.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3412//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3412//console
This message is automatically generated.
> Use direct buffers for data in write path
> -----------------------------------------
>
> Key: HDFS-3529
> URL: https://issues.apache.org/jira/browse/HDFS-3529
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 2.0.0-alpha
> Reporter: Todd Lipcon
> Assignee: Trevor Robinson
> Attachments: HDFS-3529.patch
>
>
> The write path currently makes several unnecessary data copies in order to go to and from byte arrays. We can improve performance by using direct byte buffers to avoid the copy. This is also a prerequisite for native checksum calculation (HDFS-3528)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3529) Use direct buffers for data in write
path
Posted by "Trevor Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Trevor Robinson updated HDFS-3529:
----------------------------------
Attachment: dfsio-x86-trunk-vs-3529.png
Attached image shows Ganglia charts from two runs of TestDFSIO (each with Write followed by Read): first is with this patch, second is without it. The I/O rates are essentially the same (disk-bound), but the write phase shows 5% lower CPU utilization with the patch. CPU used is a Sandy Bridge E3-1240 (4 cores, 3.3 GHz).
> Use direct buffers for data in write path
> -----------------------------------------
>
> Key: HDFS-3529
> URL: https://issues.apache.org/jira/browse/HDFS-3529
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 2.0.0-alpha
> Reporter: Todd Lipcon
> Assignee: Trevor Robinson
> Attachments: dfsio-x86-trunk-vs-3529.png, HDFS-3529.patch
>
>
> The write path currently makes several unnecessary data copies in order to go to and from byte arrays. We can improve performance by using direct byte buffers to avoid the copy. This is also a prerequisite for native checksum calculation (HDFS-3528)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write
path
Posted by "Trevor Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492024#comment-13492024 ]
Trevor Robinson commented on HDFS-3529:
---------------------------------------
My patch should enable native CRC on the datanode side, so it sounds like we just need to enable it on the client side too. This patch seemed large enough already that I wanted to get it reviewed and committed before embarking on the next step.
FYI, I swapped 4 HDDs for 4 SSDs in the E3-1240 system I mentioned earlier. On trunk, I get a TestDFSIO write throughput of 257 MiB/s (96 1.5 GiB files, 4 mappers, 1 node); with the patch, I get 326 MiB/s (a speedup of 27%). I don't think people generally run Hadoop on SSD, but just wanted to point out that this patch alone can be significant in some configurations.
> Use direct buffers for data in write path
> -----------------------------------------
>
> Key: HDFS-3529
> URL: https://issues.apache.org/jira/browse/HDFS-3529
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 2.0.0-alpha
> Reporter: Todd Lipcon
> Assignee: Trevor Robinson
> Attachments: dfsio-x86-trunk-vs-3529.png, HDFS-3529.patch
>
>
> The write path currently makes several unnecessary data copies in order to go to and from byte arrays. We can improve performance by using direct byte buffers to avoid the copy. This is also a prerequisite for native checksum calculation (HDFS-3528)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write
path
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491789#comment-13491789 ]
Todd Lipcon commented on HDFS-3529:
-----------------------------------
Hey Gopal, Trevor,
I agree we need the native CRC stuff to make a substantial savings here. I don't think the direct buffers alone are significant enough to make a noticeable difference. I had a patch in flight a few months ago which Trevor was working off of - I'll see if I can find some time to revive it on top of Trevor's latest work.
> Use direct buffers for data in write path
> -----------------------------------------
>
> Key: HDFS-3529
> URL: https://issues.apache.org/jira/browse/HDFS-3529
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 2.0.0-alpha
> Reporter: Todd Lipcon
> Assignee: Trevor Robinson
> Attachments: dfsio-x86-trunk-vs-3529.png, HDFS-3529.patch
>
>
> The write path currently makes several unnecessary data copies in order to go to and from byte arrays. We can improve performance by using direct byte buffers to avoid the copy. This is also a prerequisite for native checksum calculation (HDFS-3528)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3529) Use direct buffers for data in write
path
Posted by "Trevor Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Trevor Robinson updated HDFS-3529:
----------------------------------
Attachment: HDFS-3529.patch
The attached patch is based on [Todd Lipcon's patch|https://github.com/toddlipcon/hadoop-common/tree/trunk-write-pipeline-fast], but was modified significantly to pass all unit tests and to merge with the datanode encryption (HDFS-3637) changes.
By switching to direct buffers, the use of native CRC is automatically enabled in {{DataChecksum.verifyChunkedSums}} (HDFS-3528).
On ARM systems, I consistently see about a 10% improvement in TestDFSIO write throughput; on x86, it varies more, but the average so far is 4%. Read throughput seemed slightly higher, but was within the run-to-run deviation. Obviously, hardware and tuning configurations vary, so hopefully others will try it out and share their results. The patch is based on trunk revision {{ce25c352c5a4e69baf39fe00abdd48c832402cf3}}.
> Use direct buffers for data in write path
> -----------------------------------------
>
> Key: HDFS-3529
> URL: https://issues.apache.org/jira/browse/HDFS-3529
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 2.0.0-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: HDFS-3529.patch
>
>
> The write path currently makes several unnecessary data copies in order to go to and from byte arrays. We can improve performance by using direct byte buffers to avoid the copy. This is also a prerequisite for native checksum calculation (HDFS-3528)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write
path
Posted by "Gopal V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491686#comment-13491686 ]
Gopal V commented on HDFS-3529:
-------------------------------
I ran through some benchmarks with this patch on (and no tests failed on test-patch).
On a spinning disk it barely made any difference in execution time, but it does speed up DFS write throughput by a couple of percentage points when I backed HDFS with an SSD with I/O elevators on deadline.
But for my benchmarks, the bottleneck for the operations are on the client right now - the client DataStreamer could not write enough data to saturate the SSD on a single thread.
Adding a native chunked checksum generator similar to HDFS-3528 (well, generate is just verify without a check) on the client code would be worthwhile if this patch needs to actually improve the big picture of execution - otherwise it might only benefit the replication scenarios slightly.
> Use direct buffers for data in write path
> -----------------------------------------
>
> Key: HDFS-3529
> URL: https://issues.apache.org/jira/browse/HDFS-3529
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 2.0.0-alpha
> Reporter: Todd Lipcon
> Assignee: Trevor Robinson
> Attachments: dfsio-x86-trunk-vs-3529.png, HDFS-3529.patch
>
>
> The write path currently makes several unnecessary data copies in order to go to and from byte arrays. We can improve performance by using direct byte buffers to avoid the copy. This is also a prerequisite for native checksum calculation (HDFS-3528)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3529) Use direct buffers for data in write
path
Posted by "Trevor Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Trevor Robinson reassigned HDFS-3529:
-------------------------------------
Assignee: Trevor Robinson (was: Todd Lipcon)
> Use direct buffers for data in write path
> -----------------------------------------
>
> Key: HDFS-3529
> URL: https://issues.apache.org/jira/browse/HDFS-3529
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, performance
> Affects Versions: 2.0.0-alpha
> Reporter: Todd Lipcon
> Assignee: Trevor Robinson
> Attachments: HDFS-3529.patch
>
>
> The write path currently makes several unnecessary data copies in order to go to and from byte arrays. We can improve performance by using direct byte buffers to avoid the copy. This is also a prerequisite for native checksum calculation (HDFS-3528)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira