You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Benoit Sigoure (JIRA)" <ji...@apache.org> on 2010/10/15 10:25:32 UTC
[jira] Created: (HBASE-3115) HBaseClient wastes 1 TCP packet per
RPC
HBaseClient wastes 1 TCP packet per RPC
---------------------------------------
Key: HBASE-3115
URL: https://issues.apache.org/jira/browse/HBASE-3115
Project: HBase
Issue Type: Improvement
Components: client
Affects Versions: 0.89.20100621, 0.20.6, 0.20.5, 0.20.4, 0.20.3, 0.20.2, 0.20.1
Reporter: Benoit Sigoure
Priority: Minor
In {{ipc/HBaseClient.java}}, the method {{sendParam}} does:
{code}
out.writeInt(dataLength); //first put the data length
out.write(data, 0, dataLength);//write the data
{code}
While analyzing some tcpdump traces tonight, I saw that this consistently translates to 1 TCP packet with a 4 byte payload followed by another TCP packet with the RPC itself. This makes inefficient use of network resources and adversely affects TCP throughput. I believe each of those lines translates to a {{write}} system call on the socket's file descriptor (unnecessary system calls are also bad for performance). The code attempts to call {{out.flush();}} but this approach is ineffective on sockets in Java (as far as I empirically noticed over the past few months).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3115) HBaseClient wastes 1 TCP packet per
RPC
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922391#action_12922391 ]
Todd Lipcon commented on HBASE-3115:
------------------------------------
FYI I didn't verify with tshark or anything that this actually reduced # of packets, just kind of assuming. Did either of you guys try it out?
> HBaseClient wastes 1 TCP packet per RPC
> ---------------------------------------
>
> Key: HBASE-3115
> URL: https://issues.apache.org/jira/browse/HBASE-3115
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.20.5, 0.20.6, 0.89.20100621
> Reporter: Benoit Sigoure
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hbase-3115.txt
>
>
> In {{ipc/HBaseClient.java}}, the method {{sendParam}} does:
> {code}
> out.writeInt(dataLength); //first put the data length
> out.write(data, 0, dataLength);//write the data
> {code}
> While analyzing some tcpdump traces tonight, I saw that this consistently translates to 1 TCP packet with a 4 byte payload followed by another TCP packet with the RPC itself. This makes inefficient use of network resources and adversely affects TCP throughput. I believe each of those lines translates to a {{write}} system call on the socket's file descriptor (unnecessary system calls are also bad for performance). The code attempts to call {{out.flush();}} but this approach is ineffective on sockets in Java (as far as I empirically noticed over the past few months).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3115) HBaseClient wastes 1 TCP packet per
RPC
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922710#action_12922710 ]
Andrew Purtell commented on HBASE-3115:
---------------------------------------
Backported to 0.20 branch (if there's ever a 0.20.7)
> HBaseClient wastes 1 TCP packet per RPC
> ---------------------------------------
>
> Key: HBASE-3115
> URL: https://issues.apache.org/jira/browse/HBASE-3115
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.20.5, 0.20.6, 0.89.20100621
> Reporter: Benoit Sigoure
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.90.0
>
> Attachments: hbase-3115.txt
>
>
> In {{ipc/HBaseClient.java}}, the method {{sendParam}} does:
> {code}
> out.writeInt(dataLength); //first put the data length
> out.write(data, 0, dataLength);//write the data
> {code}
> While analyzing some tcpdump traces tonight, I saw that this consistently translates to 1 TCP packet with a 4 byte payload followed by another TCP packet with the RPC itself. This makes inefficient use of network resources and adversely affects TCP throughput. I believe each of those lines translates to a {{write}} system call on the socket's file descriptor (unnecessary system calls are also bad for performance). The code attempts to call {{out.flush();}} but this approach is ineffective on sockets in Java (as far as I empirically noticed over the past few months).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3115) HBaseClient wastes 1 TCP packet per
RPC
Posted by "Benoit Sigoure (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922435#action_12922435 ]
Benoit Sigoure commented on HBASE-3115:
---------------------------------------
No I haven't tried it either - I don't use this HBase client anymore, I use my asynchronous client instead.
But given that the patch changes the code to call out.write() only once, I can't imagine how this can be split in 2 write() system calls that would result in 2 packets.
> HBaseClient wastes 1 TCP packet per RPC
> ---------------------------------------
>
> Key: HBASE-3115
> URL: https://issues.apache.org/jira/browse/HBASE-3115
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.20.5, 0.20.6, 0.89.20100621
> Reporter: Benoit Sigoure
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hbase-3115.txt
>
>
> In {{ipc/HBaseClient.java}}, the method {{sendParam}} does:
> {code}
> out.writeInt(dataLength); //first put the data length
> out.write(data, 0, dataLength);//write the data
> {code}
> While analyzing some tcpdump traces tonight, I saw that this consistently translates to 1 TCP packet with a 4 byte payload followed by another TCP packet with the RPC itself. This makes inefficient use of network resources and adversely affects TCP throughput. I believe each of those lines translates to a {{write}} system call on the socket's file descriptor (unnecessary system calls are also bad for performance). The code attempts to call {{out.flush();}} but this approach is ineffective on sockets in Java (as far as I empirically noticed over the past few months).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3115) HBaseClient wastes 1 TCP packet per
RPC
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922430#action_12922430 ]
stack commented on HBASE-3115:
------------------------------
Not me (Pinging Benoit).
> HBaseClient wastes 1 TCP packet per RPC
> ---------------------------------------
>
> Key: HBASE-3115
> URL: https://issues.apache.org/jira/browse/HBASE-3115
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.20.5, 0.20.6, 0.89.20100621
> Reporter: Benoit Sigoure
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hbase-3115.txt
>
>
> In {{ipc/HBaseClient.java}}, the method {{sendParam}} does:
> {code}
> out.writeInt(dataLength); //first put the data length
> out.write(data, 0, dataLength);//write the data
> {code}
> While analyzing some tcpdump traces tonight, I saw that this consistently translates to 1 TCP packet with a 4 byte payload followed by another TCP packet with the RPC itself. This makes inefficient use of network resources and adversely affects TCP throughput. I believe each of those lines translates to a {{write}} system call on the socket's file descriptor (unnecessary system calls are also bad for performance). The code attempts to call {{out.flush();}} but this approach is ineffective on sockets in Java (as far as I empirically noticed over the past few months).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-3115) HBaseClient wastes 1 TCP packet per
RPC
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon reassigned HBASE-3115:
----------------------------------
Assignee: Todd Lipcon
> HBaseClient wastes 1 TCP packet per RPC
> ---------------------------------------
>
> Key: HBASE-3115
> URL: https://issues.apache.org/jira/browse/HBASE-3115
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.20.5, 0.20.6, 0.89.20100621
> Reporter: Benoit Sigoure
> Assignee: Todd Lipcon
> Priority: Minor
>
> In {{ipc/HBaseClient.java}}, the method {{sendParam}} does:
> {code}
> out.writeInt(dataLength); //first put the data length
> out.write(data, 0, dataLength);//write the data
> {code}
> While analyzing some tcpdump traces tonight, I saw that this consistently translates to 1 TCP packet with a 4 byte payload followed by another TCP packet with the RPC itself. This makes inefficient use of network resources and adversely affects TCP throughput. I believe each of those lines translates to a {{write}} system call on the socket's file descriptor (unnecessary system calls are also bad for performance). The code attempts to call {{out.flush();}} but this approach is ineffective on sockets in Java (as far as I empirically noticed over the past few months).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3115) HBaseClient wastes 1 TCP packet per
RPC
Posted by "Benoit Sigoure (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922223#action_12922223 ]
Benoit Sigoure commented on HBASE-3115:
---------------------------------------
+1 Todd, thanks a bunch.
> HBaseClient wastes 1 TCP packet per RPC
> ---------------------------------------
>
> Key: HBASE-3115
> URL: https://issues.apache.org/jira/browse/HBASE-3115
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.20.5, 0.20.6, 0.89.20100621
> Reporter: Benoit Sigoure
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hbase-3115.txt
>
>
> In {{ipc/HBaseClient.java}}, the method {{sendParam}} does:
> {code}
> out.writeInt(dataLength); //first put the data length
> out.write(data, 0, dataLength);//write the data
> {code}
> While analyzing some tcpdump traces tonight, I saw that this consistently translates to 1 TCP packet with a 4 byte payload followed by another TCP packet with the RPC itself. This makes inefficient use of network resources and adversely affects TCP throughput. I believe each of those lines translates to a {{write}} system call on the socket's file descriptor (unnecessary system calls are also bad for performance). The code attempts to call {{out.flush();}} but this approach is ineffective on sockets in Java (as far as I empirically noticed over the past few months).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-3115) HBaseClient wastes 1 TCP packet per
RPC
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon resolved HBASE-3115.
--------------------------------
Resolution: Fixed
Fix Version/s: 0.90.0
Hadoop Flags: [Reviewed]
OK, I'll go with your logic. Thanks for the good find, Benoit.
> HBaseClient wastes 1 TCP packet per RPC
> ---------------------------------------
>
> Key: HBASE-3115
> URL: https://issues.apache.org/jira/browse/HBASE-3115
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.20.5, 0.20.6, 0.89.20100621
> Reporter: Benoit Sigoure
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.90.0
>
> Attachments: hbase-3115.txt
>
>
> In {{ipc/HBaseClient.java}}, the method {{sendParam}} does:
> {code}
> out.writeInt(dataLength); //first put the data length
> out.write(data, 0, dataLength);//write the data
> {code}
> While analyzing some tcpdump traces tonight, I saw that this consistently translates to 1 TCP packet with a 4 byte payload followed by another TCP packet with the RPC itself. This makes inefficient use of network resources and adversely affects TCP throughput. I believe each of those lines translates to a {{write}} system call on the socket's file descriptor (unnecessary system calls are also bad for performance). The code attempts to call {{out.flush();}} but this approach is ineffective on sockets in Java (as far as I empirically noticed over the past few months).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3115) HBaseClient wastes 1 TCP packet per
RPC
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922358#action_12922358 ]
stack commented on HBASE-3115:
------------------------------
I'm +1 too
> HBaseClient wastes 1 TCP packet per RPC
> ---------------------------------------
>
> Key: HBASE-3115
> URL: https://issues.apache.org/jira/browse/HBASE-3115
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.20.5, 0.20.6, 0.89.20100621
> Reporter: Benoit Sigoure
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hbase-3115.txt
>
>
> In {{ipc/HBaseClient.java}}, the method {{sendParam}} does:
> {code}
> out.writeInt(dataLength); //first put the data length
> out.write(data, 0, dataLength);//write the data
> {code}
> While analyzing some tcpdump traces tonight, I saw that this consistently translates to 1 TCP packet with a 4 byte payload followed by another TCP packet with the RPC itself. This makes inefficient use of network resources and adversely affects TCP throughput. I believe each of those lines translates to a {{write}} system call on the socket's file descriptor (unnecessary system calls are also bad for performance). The code attempts to call {{out.flush();}} but this approach is ineffective on sockets in Java (as far as I empirically noticed over the past few months).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3115) HBaseClient wastes 1 TCP packet per
RPC
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-3115:
-------------------------------
Attachment: hbase-3115.txt
Maybe something like this?
> HBaseClient wastes 1 TCP packet per RPC
> ---------------------------------------
>
> Key: HBASE-3115
> URL: https://issues.apache.org/jira/browse/HBASE-3115
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.20.5, 0.20.6, 0.89.20100621
> Reporter: Benoit Sigoure
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hbase-3115.txt
>
>
> In {{ipc/HBaseClient.java}}, the method {{sendParam}} does:
> {code}
> out.writeInt(dataLength); //first put the data length
> out.write(data, 0, dataLength);//write the data
> {code}
> While analyzing some tcpdump traces tonight, I saw that this consistently translates to 1 TCP packet with a 4 byte payload followed by another TCP packet with the RPC itself. This makes inefficient use of network resources and adversely affects TCP throughput. I believe each of those lines translates to a {{write}} system call on the socket's file descriptor (unnecessary system calls are also bad for performance). The code attempts to call {{out.flush();}} but this approach is ineffective on sockets in Java (as far as I empirically noticed over the past few months).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3115) HBaseClient wastes 1 TCP packet per
RPC
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921407#action_12921407 ]
stack commented on HBASE-3115:
------------------------------
Excellent analysis work Benôit.
> HBaseClient wastes 1 TCP packet per RPC
> ---------------------------------------
>
> Key: HBASE-3115
> URL: https://issues.apache.org/jira/browse/HBASE-3115
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.20.5, 0.20.6, 0.89.20100621
> Reporter: Benoit Sigoure
> Priority: Minor
>
> In {{ipc/HBaseClient.java}}, the method {{sendParam}} does:
> {code}
> out.writeInt(dataLength); //first put the data length
> out.write(data, 0, dataLength);//write the data
> {code}
> While analyzing some tcpdump traces tonight, I saw that this consistently translates to 1 TCP packet with a 4 byte payload followed by another TCP packet with the RPC itself. This makes inefficient use of network resources and adversely affects TCP throughput. I believe each of those lines translates to a {{write}} system call on the socket's file descriptor (unnecessary system calls are also bad for performance). The code attempts to call {{out.flush();}} but this approach is ineffective on sockets in Java (as far as I empirically noticed over the past few months).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.