You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2012/09/07 05:22:07 UTC
[jira] [Created] (HDFS-3898) QJM: enable TCP_NODELAY for IPC
Todd Lipcon created HDFS-3898:
---------------------------------
Summary: QJM: enable TCP_NODELAY for IPC
Key: HDFS-3898
URL: https://issues.apache.org/jira/browse/HDFS-3898
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Currently, if the size of the edits batches is larger than the MTU, it can result in 40ms delays due to interaction between nagle's algorithm and delayed ack. Enabling TCP_NODELAY on the sockets solves this issue, so we should set those configs by default for all of the QJM-related IPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3898) QJM: enable TCP_NODELAY for IPC
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450942#comment-13450942 ]
Aaron T. Myers commented on HDFS-3898:
--------------------------------------
+1, patch looks good to me.
> QJM: enable TCP_NODELAY for IPC
> -------------------------------
>
> Key: HDFS-3898
> URL: https://issues.apache.org/jira/browse/HDFS-3898
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hdfs-3898.txt, hdfs-3898.txt
>
>
> Currently, if the size of the edits batches is larger than the MTU, it can result in 40ms delays due to interaction between nagle's algorithm and delayed ack. Enabling TCP_NODELAY on the sockets solves this issue, so we should set those configs by default for all of the QJM-related IPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3898) QJM: enable TCP_NODELAY for IPC
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3898:
------------------------------
Attachment: hdfs-3898.txt
> QJM: enable TCP_NODELAY for IPC
> -------------------------------
>
> Key: HDFS-3898
> URL: https://issues.apache.org/jira/browse/HDFS-3898
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hdfs-3898.txt
>
>
> Currently, if the size of the edits batches is larger than the MTU, it can result in 40ms delays due to interaction between nagle's algorithm and delayed ack. Enabling TCP_NODELAY on the sockets solves this issue, so we should set those configs by default for all of the QJM-related IPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3898) QJM: enable TCP_NODELAY for IPC
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450904#comment-13450904 ]
Todd Lipcon commented on HDFS-3898:
-----------------------------------
Right, Nagle's algorithm applies to all packets. But the bad (40ms-latency-causing) interaction between Nagle's Algorithm and Delayed ACK only applies when you have an even number of packets in a message (I think? or is it odd? it's complicated and I can't ever remember). The following page (coincidentally by the author of my favorite computer game ever) is pretty good at describing the issue: http://www.stuartcheshire.org/papers/NagleDelayedAck/
> QJM: enable TCP_NODELAY for IPC
> -------------------------------
>
> Key: HDFS-3898
> URL: https://issues.apache.org/jira/browse/HDFS-3898
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hdfs-3898.txt
>
>
> Currently, if the size of the edits batches is larger than the MTU, it can result in 40ms delays due to interaction between nagle's algorithm and delayed ack. Enabling TCP_NODELAY on the sockets solves this issue, so we should set those configs by default for all of the QJM-related IPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3898) QJM: enable TCP_NODELAY for IPC
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450878#comment-13450878 ]
Colin Patrick McCabe commented on HDFS-3898:
--------------------------------------------
Looks good to me.
Just one small nit. Nagle can trigger even when writing less than the MTU, though, right? I thought its primary use was coalescing lots of small packets into fewer big ones. The comment seems to imply that this is only a problem if you're writing more than the MTU in a batch, but it seems like Nagle could be a problem even if you don't do this.
> QJM: enable TCP_NODELAY for IPC
> -------------------------------
>
> Key: HDFS-3898
> URL: https://issues.apache.org/jira/browse/HDFS-3898
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hdfs-3898.txt
>
>
> Currently, if the size of the edits batches is larger than the MTU, it can result in 40ms delays due to interaction between nagle's algorithm and delayed ack. Enabling TCP_NODELAY on the sockets solves this issue, so we should set those configs by default for all of the QJM-related IPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3898) QJM: enable TCP_NODELAY for IPC
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451130#comment-13451130 ]
Colin Patrick McCabe commented on HDFS-3898:
--------------------------------------------
Thanks for the explanation. I didn't realize there was that kind of interaction between Nagle and delayed ACK. Looks good.
> QJM: enable TCP_NODELAY for IPC
> -------------------------------
>
> Key: HDFS-3898
> URL: https://issues.apache.org/jira/browse/HDFS-3898
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hdfs-3898.txt, hdfs-3898.txt
>
>
> Currently, if the size of the edits batches is larger than the MTU, it can result in 40ms delays due to interaction between nagle's algorithm and delayed ack. Enabling TCP_NODELAY on the sockets solves this issue, so we should set those configs by default for all of the QJM-related IPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3898) QJM: enable TCP_NODELAY for IPC
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon resolved HDFS-3898.
-------------------------------
Resolution: Fixed
Fix Version/s: QuorumJournalManager (HDFS-3077)
Hadoop Flags: Reviewed
Committed to branch, thanks for the reviews
> QJM: enable TCP_NODELAY for IPC
> -------------------------------
>
> Key: HDFS-3898
> URL: https://issues.apache.org/jira/browse/HDFS-3898
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: QuorumJournalManager (HDFS-3077)
>
> Attachments: hdfs-3898.txt, hdfs-3898.txt
>
>
> Currently, if the size of the edits batches is larger than the MTU, it can result in 40ms delays due to interaction between nagle's algorithm and delayed ack. Enabling TCP_NODELAY on the sockets solves this issue, so we should set those configs by default for all of the QJM-related IPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3898) QJM: enable TCP_NODELAY for IPC
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450301#comment-13450301 ]
Todd Lipcon commented on HDFS-3898:
-----------------------------------
This patch has a simple performance test which writes 1000 8kb edits. Prior to the patch, the average time taken was 40ms per edit due to nagling. After the patch, it's <2ms.
> QJM: enable TCP_NODELAY for IPC
> -------------------------------
>
> Key: HDFS-3898
> URL: https://issues.apache.org/jira/browse/HDFS-3898
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hdfs-3898.txt
>
>
> Currently, if the size of the edits batches is larger than the MTU, it can result in 40ms delays due to interaction between nagle's algorithm and delayed ack. Enabling TCP_NODELAY on the sockets solves this issue, so we should set those configs by default for all of the QJM-related IPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3898) QJM: enable TCP_NODELAY for IPC
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3898:
------------------------------
Attachment: hdfs-3898.txt
Rebased after the security fixes went in (some minor trivial conflicts)
> QJM: enable TCP_NODELAY for IPC
> -------------------------------
>
> Key: HDFS-3898
> URL: https://issues.apache.org/jira/browse/HDFS-3898
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hdfs-3898.txt, hdfs-3898.txt
>
>
> Currently, if the size of the edits batches is larger than the MTU, it can result in 40ms delays due to interaction between nagle's algorithm and delayed ack. Enabling TCP_NODELAY on the sockets solves this issue, so we should set those configs by default for all of the QJM-related IPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira