You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2012/10/15 07:19:02 UTC
[jira] [Created] (HDFS-4049) hflush performance regression due to
nagling delays
Todd Lipcon created HDFS-4049:
---------------------------------
Summary: hflush performance regression due to nagling delays
Key: HDFS-4049
URL: https://issues.apache.org/jira/browse/HDFS-4049
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node, performance
Affects Versions: 2.0.2-alpha, 3.0.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Critical
HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-4049:
------------------------------
Resolution: Fixed
Fix Version/s: 2.0.3-alpha
3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to branch-2 and trunk.
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: hdfs-4049.txt, hdfs-4049.txt, hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-4049:
------------------------------
Attachment: hdfs-4049.txt
Attached patch should fix the issue. In my test, it dropped the test case from average hflush latency of 40ms to <2ms, when running with 3 DNs.
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-4049:
------------------------------
Attachment: hdfs-4049.txt
Thanks for the review. Good catches above. New patch addresses all.
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-4049.txt, hdfs-4049.txt, hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476614#comment-13476614 ]
Todd Lipcon commented on HDFS-4049:
-----------------------------------
The failed test is due to HDFS-3948 (unrelated).
Will commit this momentarily. Thanks for the reviews, Chris, Colin and Aaron.
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-4049.txt, hdfs-4049.txt, hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476560#comment-13476560 ]
Hadoop QA commented on HDFS-4049:
---------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12549209/hdfs-4049.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHDFS
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3335//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3335//console
This message is automatically generated.
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-4049.txt, hdfs-4049.txt, hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476435#comment-13476435 ]
Aaron T. Myers commented on HDFS-4049:
--------------------------------------
Patch looks really good to me. A few little comments on the test code. +1 once these are addressed:
# In TestMultiThreadedHflush, the repl 1 test is in fact also using repl 3:
{code}
+ public void testMultipleHflushersRepl1() throws Exception {
+ doTestMultipleHflushers(3);
+ }
+
+ @Test
+ public void testMultipleHflushersRepl3() throws Exception {
+ doTestMultipleHflushers(3);
+ }
{code}
# I recommend changing the formal parameter name to "replication" or "numDatanodes" instead of "i" in the following:
{code}
+ private void doTestMultipleHflushers(int i) throws Exception {
{code}
# Not changed by your patch, but I think that the constants TestMultiThreadedHflush.numBlocks and TestMultiThreadedHflush.fileSize are unused. Mind removing them?
In addition to the code review, I also tested the patch on a 4 node physical cluster and confirmed it worked as expected.
Prior to HDFS-3721:
{noformat}
Finished in 58757ms
Latency quantiles (in microseconds):
50.00 %ile +/- 5.00%: 847
75.00 %ile +/- 2.50%: 954
90.00 %ile +/- 1.00%: 1074
95.00 %ile +/- 0.50%: 1203
99.00 %ile +/- 0.10%: 2073
{noformat}
Post HDFS-3721:
{noformat}
Finished in 1032220ms
Latency quantiles (in microseconds):
50.00 %ile +/- 5.00%: 1756
75.00 %ile +/- 2.50%: 41004
90.00 %ile +/- 1.00%: 41231
95.00 %ile +/- 0.50%: 41400
99.00 %ile +/- 0.10%: 42684
{noformat}
With the latest patch included here:
{noformat}
Finished in 58531ms
Latency quantiles (in microseconds):
50.00 %ile +/- 5.00%: 864
75.00 %ile +/- 2.50%: 970
90.00 %ile +/- 1.00%: 1096
95.00 %ile +/- 0.50%: 1237
99.00 %ile +/- 0.10%: 2131
{noformat}
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-4049.txt, hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Chris Nauroth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476252#comment-13476252 ]
Chris Nauroth commented on HDFS-4049:
-------------------------------------
Looks good. In the following lines, there is some repetition of the length calculations:
{code}
+ curPacketBuf.position(PacketHeader.PKT_LENGTHS_LEN + headerLen);
+ curPacketBuf.limit(PacketHeader.PKT_LENGTHS_LEN + headerLen + checksumsLen);
+ curPacketBuf.position(
+ PacketHeader.PKT_LENGTHS_LEN + headerLen + checksumsLen);
+ curPacketBuf.limit(
+ PacketHeader.PKT_LENGTHS_LEN + headerLen + checksumsLen + dataLen);
+ curPacketBuf.limit(
+ PacketHeader.PKT_LENGTHS_LEN + headerLen + checksumsLen + dataLen);
{code}
The method might read/maintain better if those calculations were captured in variables, like lenThroughHeader, lenThroughChecksums, and lenThroughData.
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-4049:
------------------------------
Attachment: hdfs-4049.txt
Thanks for the reviews. Addressed both the suggestions above. I also added a new unit test specifically for PacketReceiver in isolation, since some of the buffer handling is tricky.
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-4049.txt, hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475998#comment-13475998 ]
Hadoop QA commented on HDFS-4049:
---------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12549107/hdfs-4049.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3330//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3330//console
This message is automatically generated.
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476171#comment-13476171 ]
Colin Patrick McCabe commented on HDFS-4049:
--------------------------------------------
Looks good. It makes sense to use only one buffer for this.
It looks like Java doesn't expose {{writev}} in this case, which is what we would need to write multiple buffers as one packet with {{TCP_NODELAY}}. (Although there is a multi-buffer write method in {{FileChannel}}, I didn't find one for {{DataOutputStream}}).
{code}
@@ -100,12 +102,17 @@ public void run() {
}
} catch (Throwable t) {
thrown.compareAndSet(null, t);
+ } finally {
+
}
}
{code}
Extra code?
Does it make sense to add a test for the numDataNodes = 1, repl = 1 case?
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-4049:
------------------------------
Status: Patch Available (was: Open)
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 2.0.2-alpha, 3.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Chris Nauroth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476352#comment-13476352 ]
Chris Nauroth commented on HDFS-4049:
-------------------------------------
+1
Thanks, Todd. The additional comments and tests are very helpful too.
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-4049.txt, hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476450#comment-13476450 ]
Aaron T. Myers commented on HDFS-4049:
--------------------------------------
+1 pending Jenkins.
Thanks a lot, Todd.
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-4049.txt, hdfs-4049.txt, hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476401#comment-13476401 ]
Hadoop QA commented on HDFS-4049:
---------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12549185/hdfs-4049.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3333//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3333//console
This message is automatically generated.
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-4049.txt, hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476365#comment-13476365 ]
Todd Lipcon commented on HDFS-4049:
-----------------------------------
BTW, the latency readout in the benchmark currently isn't very useful since SampleQuantiles doesn't have a toString(). I hacked it locally for the purposes of running the test, and just filed HADOOP-8929 to actually get that into the code.
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-4049.txt, hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476622#comment-13476622 ]
Hudson commented on HDFS-4049:
------------------------------
Integrated in Hadoop-trunk-Commit #2862 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/2862/])
HDFS-4049. Fix hflush performance regression due to nagling delays. Contributed by Todd Lipcon. (Revision 1398591)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398591
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PacketReceiver.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer/TestPacketReceiver.java
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: hdfs-4049.txt, hdfs-4049.txt, hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477865#comment-13477865 ]
Hudson commented on HDFS-4049:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1228 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1228/])
HDFS-4049. Fix hflush performance regression due to nagling delays. Contributed by Todd Lipcon. (Revision 1398591)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398591
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PacketReceiver.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer/TestPacketReceiver.java
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: hdfs-4049.txt, hdfs-4049.txt, hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4049) hflush performance regression due to
nagling delays
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476902#comment-13476902 ]
Hudson commented on HDFS-4049:
------------------------------
Integrated in Hadoop-Yarn-trunk #5 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/5/])
HDFS-4049. Fix hflush performance regression due to nagling delays. Contributed by Todd Lipcon. (Revision 1398591)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398591
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PacketReceiver.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer/TestPacketReceiver.java
> hflush performance regression due to nagling delays
> ---------------------------------------------------
>
> Key: HDFS-4049
> URL: https://issues.apache.org/jira/browse/HDFS-4049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, performance
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: hdfs-4049.txt, hdfs-4049.txt, hdfs-4049.txt
>
>
> HDFS-3721 reworked the way that packets are mirrored through the pipeline in the datanode. This caused two write() calls where there used to be one, which interacts badly with nagling so that there are 40ms bubbles on hflush() calls. We didn't notice this in the tests because the hflush perf test only uses a single datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira