You are viewing a plain text version of this content. The canonical link for it is here.
Posted to proton@qpid.apache.org by "Affan Dar (JIRA)" <ji...@apache.org> on 2012/11/18 20:44:57 UTC
[jira] [Created] (PROTON-144) Reduce byte overhead for small
payloads
Affan Dar created PROTON-144:
--------------------------------
Summary: Reduce byte overhead for small payloads
Key: PROTON-144
URL: https://issues.apache.org/jira/browse/PROTON-144
Project: Qpid Proton
Issue Type: Bug
Components: proton-c
Affects Versions: 0.2, 0.3
Reporter: Affan Dar
Fix For: 0.2
In constrained bandwidth scenarios (e.g. on a cellular data network) any byte overhead is very costly.
>From the traces for a simple app, we are seeing a large overhead (>100 bytes) in the message payload for sending a two byte message. It seems like there are some default properties like the to and reply-to addresses that the proton client stamps onto a message and also there is padding on the actual two byte payload itself.
To be able to successfully embed the proton lib in such resource constrained devices the byte overhead needs to be trimmed down as much as the protocol allows.
---------------------------
Details of test app
---------------------------
The testing environment is OpenSUSE 11.4 64bit, the AMQP library is from SVN updated usually once a day although commits have slowed since the push for 0.2 and libopenssl version 1.0.0e-34.17.1.
The debugging output is all from proton, to get the debugging output set the Env variable "PN_TRACE_FRM=1"
The test is done connecting to localhost and sending a message across as simply as possible:
client:
pn_messenger_t *messenger = pn_messenger("b");
pn_messenger_start(messenger);
pn_message_t *message = pn_message();
pn_message_set_address(message, "amqps://0.0.0.0/a");
char data[2] = {(unsigned char)0xff, (unsigned char)0xff};
pn_message_load_data(message, data, 2);
pn_messenger_put(messenger, message);
pn_messenger_send(messenger);
server code:
pn_messenger_t *messenger = pn_messenger("a");
pn_messenger_start(messenger);
pn_messenger_subscribe(messenger, "amqps://~0.0.0.0");
pn_message_t *message = pn_message();
pn_messenger_recv(messenger, 1);
pn_messenger_get(messenger, message);
size_t size = 2;
char data[2];
pn_message_save_data(message, data, &size);
server output:
Accepted from localhost.localdomain:36331
-> SASL
[0x622180:0] -> SASL-MECHANISMS @64 [@PN_SYMBOL[:ANONYMOUS]]
[0x622180:0] -> SASL-OUTCOME @68 [0]
-> AMQP
[0x61c7c0:0] -> OPEN @16 ["a", null, null, null, null, null, null, null, null]
<- SASL
[0x622180:0] <- SASL-INIT @65 [:ANONYMOUS, b""]
<- AMQP
[0x61c7c0:0] <- OPEN @16 ["b", "0.0.0.0", null, null, null, null, null, null, null]
[0x61c7c0:1] <- BEGIN @17 [null, 0, 1024, 1024]
[0x61c7c0:1] <- ATTACH @18 ["sender-xxx", 1, false, null, null, @40 ["a", 0, null, 0, false, null, null, null, null, null, null], @41 ["a", 0, null, 0, false, null, null], null, null, 0]
[0x61c7c0:1] -> BEGIN @17 [1, 0, 1024, 1024]
[0x61c7c0:1] -> ATTACH @18 ["sender-xxx", 1, true, null, null, @40 ["a", 0, null, 0, false, null, null, null, null, null, null], @41 ["a", 0, null, 0, false, null, null], null, null, 0]
[0x61c7c0:1] -> FLOW @19 [0, 1024, 0, 1024, 1, 0, 1, null, false]
[0x61c7c0:1] <- TRANSFER @20 [1, 0, b"\x00\x00\x00\x00\x00\x00\x00\x00", 0, true, false] (133) "\x00\x80\x00\x00\x00\x00\x00\x00\x00p\xd0\x00\x00\x00\x10\x00\x00\x00\x05V\x00P\x04@V\x00p\x00\x00\x00\x00\x00\x80\x00\x00\x00\x00\x00\x00\x00s\xd0\x00\x00\x00F\x00\x00\x00\x0d@@\xb1\x00\x00\x00\x11amqps://0.0.0.0/a@\xb1\x00\x00\x00\x08amqp://b@@@\x83\x00\x00\x00\x00\x00\x00\x00\x00\x83\x00\x00\x00\x00\x00\x00\x00\x00@p\x00\x00\x00\x00@\x00\x80\x00\x00\x00\x00\x00\x00\x00w\xb0\x00\x00\x00\x02\xff\xff"
client output:
Connected to 0.0.0.0:5671
-> SASL
[0x620020:0] -> SASL-INIT @65 [:ANONYMOUS, b""]
<- SASL
[0x620020:0] <- SASL-MECHANISMS @64 [@PN_SYMBOL[:ANONYMOUS]]
[0x620020:0] <- SASL-OUTCOME @68 [0]
<- AMQP
[0x61a840:0] <- OPEN @16 ["a", null, null, null, null, null, null, null, null]
-> AMQP
[0x61a840:0] -> OPEN @16 ["b", "0.0.0.0", null, null, null, null, null, null, null]
[0x61a840:1] -> BEGIN @17 [null, 0, 1024, 1024]
[0x61a840:1] -> ATTACH @18 ["sender-xxx", 1, false, null, null, @40 ["a", 0, null, 0, false, null, null, null, null, null, null], @41 ["a", 0, null, 0, false, null, null], null, null, 0]
[0x61a840:1] <- BEGIN @17 [1, 0, 1024, 1024]
[0x61a840:1] <- ATTACH @18 ["sender-xxx", 1, true, null, null, @40 ["a", 0, null, 0, false, null, null, null, null, null, null], @41 ["a", 0, null, 0, false, null, null], null, null, 0]
[0x61a840:1] <- FLOW @19 [0, 1024, 0, 1024, 1, 0, 1, null, false]
[0x61a840:1] -> TRANSFER @20 [1, 0, b"\x00\x00\x00\x00\x00\x00\x00\x00", 0, true, false] (133) "\x00\x80\x00\x00\x00\x00\x00\x00\x00p\xd0\x00\x00\x00\x10\x00\x00\x00\x05V\x00P\x04@V\x00p\x00\x00\x00\x00\x00\x80\x00\x00\x00\x00\x00\x00\x00s\xd0\x00\x00\x00F\x00\x00\x00\x0d@@\xb1\x00\x00\x00\x11amqps://0.0.0.0/a@\xb1\x00\x00\x00\x08amqp://b@@@\x83\x00\x00\x00\x00\x00\x00\x00\x00\x83\x00\x00\x00\x00\x00\x00\x00\x00@p\x00\x00\x00\x00@\x00\x80\x00\x00\x00\x00\x00\x00\x00w\xb0\x00\x00\x00\x02\xff\xff
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PROTON-144) Reduce byte overhead for small
payloads
Posted by "Ted Ross (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PROTON-144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500270#comment-13500270 ]
Ted Ross commented on PROTON-144:
---------------------------------
Some more thoughts...
Even using a stripped-down, pre-settled (fire/forget) delivery mode, the endpoints are going to generate flow frames going back to the sender. This is additional overhead. Flow-frame overhead could be significantly reduced by batching (i.e. issuing credit not one at a time, but in relatively large batches).
This is already under the control of an application written against the engine API. Messenger would require modifications (I think) to support this.
> Reduce byte overhead for small payloads
> ---------------------------------------
>
> Key: PROTON-144
> URL: https://issues.apache.org/jira/browse/PROTON-144
> Project: Qpid Proton
> Issue Type: Bug
> Components: proton-c
> Affects Versions: 0.2, 0.3
> Reporter: Affan Dar
> Labels: message, performance
> Fix For: 0.2
>
>
> In constrained bandwidth scenarios (e.g. on a cellular data network) any byte overhead is very costly.
> From the traces for a simple app, we are seeing a large overhead (>100 bytes) in the message payload for sending a two byte message. It seems like there are some default properties like the to and reply-to addresses that the proton client stamps onto a message and also there is padding on the actual two byte payload itself.
> To be able to successfully embed the proton lib in such resource constrained devices the byte overhead needs to be trimmed down as much as the protocol allows.
> ---------------------------
> Details of test app
> ---------------------------
> The testing environment is OpenSUSE 11.4 64bit, the AMQP library is from SVN updated usually once a day although commits have slowed since the push for 0.2 and libopenssl version 1.0.0e-34.17.1.
> The debugging output is all from proton, to get the debugging output set the Env variable "PN_TRACE_FRM=1"
> The test is done connecting to localhost and sending a message across as simply as possible:
> client:
> pn_messenger_t *messenger = pn_messenger("b");
> pn_messenger_start(messenger);
> pn_message_t *message = pn_message();
> pn_message_set_address(message, "amqps://0.0.0.0/a");
> char data[2] = {(unsigned char)0xff, (unsigned char)0xff};
> pn_message_load_data(message, data, 2);
> pn_messenger_put(messenger, message);
> pn_messenger_send(messenger);
>
> server code:
> pn_messenger_t *messenger = pn_messenger("a");
> pn_messenger_start(messenger);
> pn_messenger_subscribe(messenger, "amqps://~0.0.0.0");
> pn_message_t *message = pn_message();
> pn_messenger_recv(messenger, 1);
> pn_messenger_get(messenger, message);
> size_t size = 2;
> char data[2];
> pn_message_save_data(message, data, &size);
>
> server output:
> Accepted from localhost.localdomain:36331
> -> SASL
> [0x622180:0] -> SASL-MECHANISMS @64 [@PN_SYMBOL[:ANONYMOUS]]
> [0x622180:0] -> SASL-OUTCOME @68 [0]
> -> AMQP
> [0x61c7c0:0] -> OPEN @16 ["a", null, null, null, null, null, null, null, null]
> <- SASL
> [0x622180:0] <- SASL-INIT @65 [:ANONYMOUS, b""]
> <- AMQP
> [0x61c7c0:0] <- OPEN @16 ["b", "0.0.0.0", null, null, null, null, null, null, null]
> [0x61c7c0:1] <- BEGIN @17 [null, 0, 1024, 1024]
> [0x61c7c0:1] <- ATTACH @18 ["sender-xxx", 1, false, null, null, @40 ["a", 0, null, 0, false, null, null, null, null, null, null], @41 ["a", 0, null, 0, false, null, null], null, null, 0]
> [0x61c7c0:1] -> BEGIN @17 [1, 0, 1024, 1024]
> [0x61c7c0:1] -> ATTACH @18 ["sender-xxx", 1, true, null, null, @40 ["a", 0, null, 0, false, null, null, null, null, null, null], @41 ["a", 0, null, 0, false, null, null], null, null, 0]
> [0x61c7c0:1] -> FLOW @19 [0, 1024, 0, 1024, 1, 0, 1, null, false]
> [0x61c7c0:1] <- TRANSFER @20 [1, 0, b"\x00\x00\x00\x00\x00\x00\x00\x00", 0, true, false] (133) "\x00\x80\x00\x00\x00\x00\x00\x00\x00p\xd0\x00\x00\x00\x10\x00\x00\x00\x05V\x00P\x04@V\x00p\x00\x00\x00\x00\x00\x80\x00\x00\x00\x00\x00\x00\x00s\xd0\x00\x00\x00F\x00\x00\x00\x0d@@\xb1\x00\x00\x00\x11amqps://0.0.0.0/a@\xb1\x00\x00\x00\x08amqp://b@@@\x83\x00\x00\x00\x00\x00\x00\x00\x00\x83\x00\x00\x00\x00\x00\x00\x00\x00@p\x00\x00\x00\x00@\x00\x80\x00\x00\x00\x00\x00\x00\x00w\xb0\x00\x00\x00\x02\xff\xff"
>
> client output:
> Connected to 0.0.0.0:5671
> -> SASL
> [0x620020:0] -> SASL-INIT @65 [:ANONYMOUS, b""]
> <- SASL
> [0x620020:0] <- SASL-MECHANISMS @64 [@PN_SYMBOL[:ANONYMOUS]]
> [0x620020:0] <- SASL-OUTCOME @68 [0]
> <- AMQP
> [0x61a840:0] <- OPEN @16 ["a", null, null, null, null, null, null, null, null]
> -> AMQP
> [0x61a840:0] -> OPEN @16 ["b", "0.0.0.0", null, null, null, null, null, null, null]
> [0x61a840:1] -> BEGIN @17 [null, 0, 1024, 1024]
> [0x61a840:1] -> ATTACH @18 ["sender-xxx", 1, false, null, null, @40 ["a", 0, null, 0, false, null, null, null, null, null, null], @41 ["a", 0, null, 0, false, null, null], null, null, 0]
> [0x61a840:1] <- BEGIN @17 [1, 0, 1024, 1024]
> [0x61a840:1] <- ATTACH @18 ["sender-xxx", 1, true, null, null, @40 ["a", 0, null, 0, false, null, null, null, null, null, null], @41 ["a", 0, null, 0, false, null, null], null, null, 0]
> [0x61a840:1] <- FLOW @19 [0, 1024, 0, 1024, 1, 0, 1, null, false]
> [0x61a840:1] -> TRANSFER @20 [1, 0, b"\x00\x00\x00\x00\x00\x00\x00\x00", 0, true, false] (133) "\x00\x80\x00\x00\x00\x00\x00\x00\x00p\xd0\x00\x00\x00\x10\x00\x00\x00\x05V\x00P\x04@V\x00p\x00\x00\x00\x00\x00\x80\x00\x00\x00\x00\x00\x00\x00s\xd0\x00\x00\x00F\x00\x00\x00\x0d@@\xb1\x00\x00\x00\x11amqps://0.0.0.0/a@\xb1\x00\x00\x00\x08amqp://b@@@\x83\x00\x00\x00\x00\x00\x00\x00\x00\x83\x00\x00\x00\x00\x00\x00\x00\x00@p\x00\x00\x00\x00@\x00\x80\x00\x00\x00\x00\x00\x00\x00w\xb0\x00\x00\x00\x02\xff\xff
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PROTON-144) Reduce byte overhead for small
payloads
Posted by "Ted Ross (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PROTON-144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500265#comment-13500265 ]
Ted Ross commented on PROTON-144:
---------------------------------
This is good input...
There are a number of ways that we could reduce the overhead for messages in bandwidth-sensitive environments:
1) Proton defaults to using 64-bit performative identifiers. If 8-bit identifiers were used, the overhead would be reduced by 7 octets per performative. I see no reason why this shouldn't be the default behavior of proton.
2) Once a link is established, transfers over that link can have very low overhead (i.e. the frame header + the transfer performative). The AMQP specification states that all of the various header fields in a message are optional. It is therefore permissible to send and receive deliveries that contain only user content. The overhead in this case is around 39 octets (assuming an 8-bit performative id, a 1-octet delivery tag, and list8 format). It is possible that another 12 octets could be removed as well if 32-bit integer fields were replaced by 8-bit fields.
Reducing the delivery overhead would need to be done in proton-engine. Eliminating the message overhead can already be done with the engine API (using raw deliveries). Providing such a capability in messenger would require additional work (perhaps an ultra-low overhead fire-and-forget delivery mode).
> Reduce byte overhead for small payloads
> ---------------------------------------
>
> Key: PROTON-144
> URL: https://issues.apache.org/jira/browse/PROTON-144
> Project: Qpid Proton
> Issue Type: Bug
> Components: proton-c
> Affects Versions: 0.2, 0.3
> Reporter: Affan Dar
> Labels: message, performance
> Fix For: 0.2
>
>
> In constrained bandwidth scenarios (e.g. on a cellular data network) any byte overhead is very costly.
> From the traces for a simple app, we are seeing a large overhead (>100 bytes) in the message payload for sending a two byte message. It seems like there are some default properties like the to and reply-to addresses that the proton client stamps onto a message and also there is padding on the actual two byte payload itself.
> To be able to successfully embed the proton lib in such resource constrained devices the byte overhead needs to be trimmed down as much as the protocol allows.
> ---------------------------
> Details of test app
> ---------------------------
> The testing environment is OpenSUSE 11.4 64bit, the AMQP library is from SVN updated usually once a day although commits have slowed since the push for 0.2 and libopenssl version 1.0.0e-34.17.1.
> The debugging output is all from proton, to get the debugging output set the Env variable "PN_TRACE_FRM=1"
> The test is done connecting to localhost and sending a message across as simply as possible:
> client:
> pn_messenger_t *messenger = pn_messenger("b");
> pn_messenger_start(messenger);
> pn_message_t *message = pn_message();
> pn_message_set_address(message, "amqps://0.0.0.0/a");
> char data[2] = {(unsigned char)0xff, (unsigned char)0xff};
> pn_message_load_data(message, data, 2);
> pn_messenger_put(messenger, message);
> pn_messenger_send(messenger);
>
> server code:
> pn_messenger_t *messenger = pn_messenger("a");
> pn_messenger_start(messenger);
> pn_messenger_subscribe(messenger, "amqps://~0.0.0.0");
> pn_message_t *message = pn_message();
> pn_messenger_recv(messenger, 1);
> pn_messenger_get(messenger, message);
> size_t size = 2;
> char data[2];
> pn_message_save_data(message, data, &size);
>
> server output:
> Accepted from localhost.localdomain:36331
> -> SASL
> [0x622180:0] -> SASL-MECHANISMS @64 [@PN_SYMBOL[:ANONYMOUS]]
> [0x622180:0] -> SASL-OUTCOME @68 [0]
> -> AMQP
> [0x61c7c0:0] -> OPEN @16 ["a", null, null, null, null, null, null, null, null]
> <- SASL
> [0x622180:0] <- SASL-INIT @65 [:ANONYMOUS, b""]
> <- AMQP
> [0x61c7c0:0] <- OPEN @16 ["b", "0.0.0.0", null, null, null, null, null, null, null]
> [0x61c7c0:1] <- BEGIN @17 [null, 0, 1024, 1024]
> [0x61c7c0:1] <- ATTACH @18 ["sender-xxx", 1, false, null, null, @40 ["a", 0, null, 0, false, null, null, null, null, null, null], @41 ["a", 0, null, 0, false, null, null], null, null, 0]
> [0x61c7c0:1] -> BEGIN @17 [1, 0, 1024, 1024]
> [0x61c7c0:1] -> ATTACH @18 ["sender-xxx", 1, true, null, null, @40 ["a", 0, null, 0, false, null, null, null, null, null, null], @41 ["a", 0, null, 0, false, null, null], null, null, 0]
> [0x61c7c0:1] -> FLOW @19 [0, 1024, 0, 1024, 1, 0, 1, null, false]
> [0x61c7c0:1] <- TRANSFER @20 [1, 0, b"\x00\x00\x00\x00\x00\x00\x00\x00", 0, true, false] (133) "\x00\x80\x00\x00\x00\x00\x00\x00\x00p\xd0\x00\x00\x00\x10\x00\x00\x00\x05V\x00P\x04@V\x00p\x00\x00\x00\x00\x00\x80\x00\x00\x00\x00\x00\x00\x00s\xd0\x00\x00\x00F\x00\x00\x00\x0d@@\xb1\x00\x00\x00\x11amqps://0.0.0.0/a@\xb1\x00\x00\x00\x08amqp://b@@@\x83\x00\x00\x00\x00\x00\x00\x00\x00\x83\x00\x00\x00\x00\x00\x00\x00\x00@p\x00\x00\x00\x00@\x00\x80\x00\x00\x00\x00\x00\x00\x00w\xb0\x00\x00\x00\x02\xff\xff"
>
> client output:
> Connected to 0.0.0.0:5671
> -> SASL
> [0x620020:0] -> SASL-INIT @65 [:ANONYMOUS, b""]
> <- SASL
> [0x620020:0] <- SASL-MECHANISMS @64 [@PN_SYMBOL[:ANONYMOUS]]
> [0x620020:0] <- SASL-OUTCOME @68 [0]
> <- AMQP
> [0x61a840:0] <- OPEN @16 ["a", null, null, null, null, null, null, null, null]
> -> AMQP
> [0x61a840:0] -> OPEN @16 ["b", "0.0.0.0", null, null, null, null, null, null, null]
> [0x61a840:1] -> BEGIN @17 [null, 0, 1024, 1024]
> [0x61a840:1] -> ATTACH @18 ["sender-xxx", 1, false, null, null, @40 ["a", 0, null, 0, false, null, null, null, null, null, null], @41 ["a", 0, null, 0, false, null, null], null, null, 0]
> [0x61a840:1] <- BEGIN @17 [1, 0, 1024, 1024]
> [0x61a840:1] <- ATTACH @18 ["sender-xxx", 1, true, null, null, @40 ["a", 0, null, 0, false, null, null, null, null, null, null], @41 ["a", 0, null, 0, false, null, null], null, null, 0]
> [0x61a840:1] <- FLOW @19 [0, 1024, 0, 1024, 1, 0, 1, null, false]
> [0x61a840:1] -> TRANSFER @20 [1, 0, b"\x00\x00\x00\x00\x00\x00\x00\x00", 0, true, false] (133) "\x00\x80\x00\x00\x00\x00\x00\x00\x00p\xd0\x00\x00\x00\x10\x00\x00\x00\x05V\x00P\x04@V\x00p\x00\x00\x00\x00\x00\x80\x00\x00\x00\x00\x00\x00\x00s\xd0\x00\x00\x00F\x00\x00\x00\x0d@@\xb1\x00\x00\x00\x11amqps://0.0.0.0/a@\xb1\x00\x00\x00\x08amqp://b@@@\x83\x00\x00\x00\x00\x00\x00\x00\x00\x83\x00\x00\x00\x00\x00\x00\x00\x00@p\x00\x00\x00\x00@\x00\x80\x00\x00\x00\x00\x00\x00\x00w\xb0\x00\x00\x00\x02\xff\xff
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira