You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Michael Goulish <mg...@redhat.com> on 2021/02/11 19:08:31 UTC
Dispatch Router: Changing buffer size in buffer.c blows up AMQP.
OK, so in the file Dispatch Router file src/buffer.c I changed this:
size_t BUFFER_SIZE = 512;
to this:
size_t BUFFER_SIZE = 4096;
Gordon tells me that's like 8 times bigger.
It makes a terrific difference in throughput in the TCP adapter, and if you
limit the sender to the throughput that the receiver can accept, it can go
Real Fast with no memory bloat. ( Like 15 Gbit/sec )
But.
AMQP throughput is Not Happy with this change.
Some of the managed fields grow rapidly (although not enough to account for
total memory growth) -- and throughput gradually drops to a crawl.
Here are the fields that increase dramatically (like 10x or more) -- and
the ones that don't much change.
qd_bitmask_t
*qd_buffer_t *
qd_composed_field_t
qd_composite_t
qd_connection_t
qd_hash_handle_t
qd_hash_item_t
qd_iterator_t
*qd_link_ref_t*
qd_link_t
qd_listener_t
qd_log_entry_t
qd_management_context_t
*qd_message_content_t*
*qd_message_t*
qd_node_t
qd_parse_node_t
qd_parse_tree_t
qd_parsed_field_t
qd_session_t
qd_timer_t
*qdr_action_t*
qdr_address_config_t
qdr_address_t
qdr_connection_info_t
qdr_connection_t
qdr_connection_work_t
qdr_core_timer_t
qdr_delivery_cleanup_t
*qdr_delivery_ref_t*
*qdr_delivery_t*
qdr_field_t
qdr_general_work_t
qdr_link_ref_t
qdr_link_t
qdr_link_work_t
qdr_query_t
qdr_terminus_t
Does anyone have a great idea about any experiment I could do,
instrumentation I could add, whatever -- that might help to further
diagnose what is going on?
Re: Dispatch Router: Changing buffer size in buffer.c blows up AMQP.
Posted by Ted Ross <tr...@apache.org>.
Another thing to consider when increasing the buffer size is how to adjust
the "Q2" limit. This limits the number of buffers that will be stored in a
streaming message at any one time. By increasing the buffer size by 8x,
the Q2 limit in bytes is also increased 8x.
This won't have any effect on your small-message test, but it will affect
the router's memory consumption with the transfer of large or streaming
messages.
-Ted
On Fri, Feb 12, 2021 at 11:56 AM Ted Ross <tr...@apache.org> wrote:
>
>
> On Fri, Feb 12, 2021 at 2:44 AM Michael Goulish <mg...@redhat.com>
> wrote:
>
>>
>>
>> *Can you explain how you are measuring AMQP throughput? What message
>> sizes are you using? Credit windows? How many senders and receivers? Max
>> frame*
>> * size?*
>>
>> Oops! Good point. Describe the Test!
>>
>> 100 senders, 100 receivers, 100 unique addresses -- each sender sends to
>> one receiver.
>> Each sender is throttled to 100 messages per second (Apparently I Really
>> Like the number 100).
>> And message size is .... wait for it ... 100. (payload size .. so
>> really 139 or something like that.)
>>
>
> Though I'm not sure exactly what's causing the strange things you are
> seeing, this is not a good test to evaluate the effect of the larger buffer
> size.
>
> Since the message sizes are so small, they will be using the same number
> of buffers in both size cases (512 and 4096). The 100 byte messages fit
> into both buffer sizes. The router will not place multiple messages into
> the same buffer. So, with 512 byte buffers, this test leaves ~400 bytes
> unused per buffer. With 4096 byte buffers, it leaves ~4000 bytes unused
> per buffer. You are allocating a lot more buffer space for no benefit.
>
> A better test would involve much larger messages, maybe 64K, 128K, or more.
>
>
>>
>> Credit window is 1000.
>>
>> I can't find anything in my router config nor in my C client code about
>> max frame size. What do I get by default? Or, how can I check that?
>>
>> The way I measured throughput was that -- first -- I noticed that when I
>> made the test go longer, i.e. send 20 million total messages instead of the
>> original 1 million -- it was taking much longer than I expected. So I had
>> each receiver log a message every time its total received messages was
>> divisible by 1000.
>>
>> What I saw was that the first thousand came after 11 seconds (just about
>> as expected because of sender-throttle to 100/sec) but that later thousands
>> became slower. By the time I stopped the test -- after more than 50,000
>> messages per receiver -- each thousand was taking ... well ... look at this
>> very interesting graph that I made of one receiver's behavior.
>>
>> This graph is made by just noting the time when you receive each
>> thousandth message (time since test started) and graphing that -- so we
>> expect to see an upward-sloping straight line whose slope is determined by
>> how long it takes to receive each 1000 messages (should be close to 10
>> seconds).
>>
>> [image: messages_vs_time.jpg]
>>
>> I'm glad I graphed this! This inflection point was a total shock to me.
>> NOTE TO SELF: always graph everything from now on forever.
>>
>> I guess Something Interesting happened at about 28 seconds!
>>
>> Maybe what I need ... is a reading from "qdstat -m" just before and after
>> that inflection point !?!??
>>
>>
>>
>> On Thu, Feb 11, 2021 at 5:37 PM Ted Ross <tr...@apache.org> wrote:
>>
>>> On Thu, Feb 11, 2021 at 2:08 PM Michael Goulish <mg...@redhat.com>
>>> wrote:
>>>
>>> > OK, so in the file Dispatch Router file src/buffer.c I changed this:
>>> > size_t BUFFER_SIZE = 512;
>>> > to this:
>>> > size_t BUFFER_SIZE = 4096;
>>> >
>>> > Gordon tells me that's like 8 times bigger.
>>> >
>>> >
>>> > It makes a terrific difference in throughput in the TCP adapter, and
>>> if you
>>> > limit the sender to the throughput that the receiver can accept, it
>>> can go
>>> > Real Fast with no memory bloat. ( Like 15 Gbit/sec )
>>> >
>>> > But.
>>> > AMQP throughput is Not Happy with this change.
>>> >
>>> > Some of the managed fields grow rapidly (although not enough to
>>> account for
>>> > total memory growth) -- and throughput gradually drops to a crawl.
>>> >
>>> > Here are the fields that increase dramatically (like 10x or more) --
>>> and
>>> > the ones that don't much change.
>>> >
>>> > qd_bitmask_t
>>> > *qd_buffer_t *
>>> > qd_composed_field_t
>>> > qd_composite_t
>>> > qd_connection_t
>>> > qd_hash_handle_t
>>> > qd_hash_item_t
>>> > qd_iterator_t
>>> > *qd_link_ref_t*
>>> > qd_link_t
>>> > qd_listener_t
>>> > qd_log_entry_t
>>> > qd_management_context_t
>>> > *qd_message_content_t*
>>> > *qd_message_t*
>>> > qd_node_t
>>> > qd_parse_node_t
>>> > qd_parse_tree_t
>>> > qd_parsed_field_t
>>> > qd_session_t
>>> > qd_timer_t
>>> > *qdr_action_t*
>>> > qdr_address_config_t
>>> > qdr_address_t
>>> > qdr_connection_info_t
>>> > qdr_connection_t
>>> > qdr_connection_work_t
>>> > qdr_core_timer_t
>>> > qdr_delivery_cleanup_t
>>> > *qdr_delivery_ref_t*
>>> > *qdr_delivery_t*
>>> > qdr_field_t
>>> > qdr_general_work_t
>>> > qdr_link_ref_t
>>> > qdr_link_t
>>> > qdr_link_work_t
>>> > qdr_query_t
>>> > qdr_terminus_t
>>> >
>>> >
>>> > Does anyone have a great idea about any experiment I could do,
>>> > instrumentation I could add, whatever -- that might help to further
>>> > diagnose what is going on?
>>> >
>>>
>>> Can you explain how you are measuring AMQP throughput? What message
>>> sizes
>>> are you using? Credit windows? How many senders and receivers? Max
>>> frame
>>> size?
>>>
>>
Re: Dispatch Router: Changing buffer size in buffer.c blows up AMQP.
Posted by Ted Ross <tr...@apache.org>.
On Fri, Feb 12, 2021 at 2:44 AM Michael Goulish <mg...@redhat.com> wrote:
>
>
> *Can you explain how you are measuring AMQP throughput? What message
> sizes are you using? Credit windows? How many senders and receivers? Max
> frame*
> * size?*
>
> Oops! Good point. Describe the Test!
>
> 100 senders, 100 receivers, 100 unique addresses -- each sender sends to
> one receiver.
> Each sender is throttled to 100 messages per second (Apparently I Really
> Like the number 100).
> And message size is .... wait for it ... 100. (payload size .. so
> really 139 or something like that.)
>
Though I'm not sure exactly what's causing the strange things you are
seeing, this is not a good test to evaluate the effect of the larger buffer
size.
Since the message sizes are so small, they will be using the same number of
buffers in both size cases (512 and 4096). The 100 byte messages fit into
both buffer sizes. The router will not place multiple messages into the
same buffer. So, with 512 byte buffers, this test leaves ~400 bytes unused
per buffer. With 4096 byte buffers, it leaves ~4000 bytes unused per
buffer. You are allocating a lot more buffer space for no benefit.
A better test would involve much larger messages, maybe 64K, 128K, or more.
>
> Credit window is 1000.
>
> I can't find anything in my router config nor in my C client code about
> max frame size. What do I get by default? Or, how can I check that?
>
> The way I measured throughput was that -- first -- I noticed that when I
> made the test go longer, i.e. send 20 million total messages instead of the
> original 1 million -- it was taking much longer than I expected. So I had
> each receiver log a message every time its total received messages was
> divisible by 1000.
>
> What I saw was that the first thousand came after 11 seconds (just about
> as expected because of sender-throttle to 100/sec) but that later thousands
> became slower. By the time I stopped the test -- after more than 50,000
> messages per receiver -- each thousand was taking ... well ... look at this
> very interesting graph that I made of one receiver's behavior.
>
> This graph is made by just noting the time when you receive each
> thousandth message (time since test started) and graphing that -- so we
> expect to see an upward-sloping straight line whose slope is determined by
> how long it takes to receive each 1000 messages (should be close to 10
> seconds).
>
> [image: messages_vs_time.jpg]
>
> I'm glad I graphed this! This inflection point was a total shock to me.
> NOTE TO SELF: always graph everything from now on forever.
>
> I guess Something Interesting happened at about 28 seconds!
>
> Maybe what I need ... is a reading from "qdstat -m" just before and after
> that inflection point !?!??
>
>
>
> On Thu, Feb 11, 2021 at 5:37 PM Ted Ross <tr...@apache.org> wrote:
>
>> On Thu, Feb 11, 2021 at 2:08 PM Michael Goulish <mg...@redhat.com>
>> wrote:
>>
>> > OK, so in the file Dispatch Router file src/buffer.c I changed this:
>> > size_t BUFFER_SIZE = 512;
>> > to this:
>> > size_t BUFFER_SIZE = 4096;
>> >
>> > Gordon tells me that's like 8 times bigger.
>> >
>> >
>> > It makes a terrific difference in throughput in the TCP adapter, and if
>> you
>> > limit the sender to the throughput that the receiver can accept, it can
>> go
>> > Real Fast with no memory bloat. ( Like 15 Gbit/sec )
>> >
>> > But.
>> > AMQP throughput is Not Happy with this change.
>> >
>> > Some of the managed fields grow rapidly (although not enough to account
>> for
>> > total memory growth) -- and throughput gradually drops to a crawl.
>> >
>> > Here are the fields that increase dramatically (like 10x or more) -- and
>> > the ones that don't much change.
>> >
>> > qd_bitmask_t
>> > *qd_buffer_t *
>> > qd_composed_field_t
>> > qd_composite_t
>> > qd_connection_t
>> > qd_hash_handle_t
>> > qd_hash_item_t
>> > qd_iterator_t
>> > *qd_link_ref_t*
>> > qd_link_t
>> > qd_listener_t
>> > qd_log_entry_t
>> > qd_management_context_t
>> > *qd_message_content_t*
>> > *qd_message_t*
>> > qd_node_t
>> > qd_parse_node_t
>> > qd_parse_tree_t
>> > qd_parsed_field_t
>> > qd_session_t
>> > qd_timer_t
>> > *qdr_action_t*
>> > qdr_address_config_t
>> > qdr_address_t
>> > qdr_connection_info_t
>> > qdr_connection_t
>> > qdr_connection_work_t
>> > qdr_core_timer_t
>> > qdr_delivery_cleanup_t
>> > *qdr_delivery_ref_t*
>> > *qdr_delivery_t*
>> > qdr_field_t
>> > qdr_general_work_t
>> > qdr_link_ref_t
>> > qdr_link_t
>> > qdr_link_work_t
>> > qdr_query_t
>> > qdr_terminus_t
>> >
>> >
>> > Does anyone have a great idea about any experiment I could do,
>> > instrumentation I could add, whatever -- that might help to further
>> > diagnose what is going on?
>> >
>>
>> Can you explain how you are measuring AMQP throughput? What message sizes
>> are you using? Credit windows? How many senders and receivers? Max
>> frame
>> size?
>>
>
Re: Dispatch Router: Changing buffer size in buffer.c blows up AMQP.
Posted by Michael Goulish <mg...@redhat.com>.
*A better test would involve much larger messages, maybe 64K, 128K, or
more.*
Oh I see -- OK, that will be easy. This is something that needs to be
fixed though, I expect.
On Fri, Feb 12, 2021 at 12:08 PM Michael Goulish <mg...@redhat.com>
wrote:
> A further note.
> I just looked at all 100 receivers this way.
> They all have the same inflection point at 28000 messages.
> But ... it could also have been a particular point in time that triggered
> it -- because the all hit the 28000 message mark within 2.5 seconds of each
> other.
>
> On Fri, Feb 12, 2021 at 11:14 AM Michael Goulish <mg...@redhat.com>
> wrote:
>
>> Sorry -- I didn't realize this list would remove the image of my graph.
>>
>> Can everyone see this?
>> <https://www.dropbox.com/s/4t1xbp46y57mfgn/messages_vs_time.jpg?dl=0>
>>
>> On Fri, Feb 12, 2021 at 7:51 AM Chuck Rolke <cr...@redhat.com> wrote:
>>
>>> The mail list scrubs attachments. Maybe create a jira and add the image
>>> to that.
>>>
>>> ----- Original Message -----
>>> > From: "Michael Goulish" <mg...@redhat.com>
>>> > To: users@qpid.apache.org
>>> > Sent: Friday, February 12, 2021 2:43:40 AM
>>> > Subject: Re: Dispatch Router: Changing buffer size in buffer.c blows
>>> up AMQP.
>>> >
>>> > *Can you explain how you are measuring AMQP throughput? What message
>>> sizes
>>> > are you using? Credit windows? How many senders and receivers? Max
>>> frame*
>>> > * size?*
>>> >
>>> > Oops! Good point. Describe the Test!
>>> >
>>> > 100 senders, 100 receivers, 100 unique addresses -- each sender sends
>>> to
>>> > one receiver.
>>> > Each sender is throttled to 100 messages per second (Apparently I
>>> Really
>>> > Like the number 100).
>>> > And message size is .... wait for it ... 100. (payload size .. so
>>> > really 139 or something like that.)
>>> >
>>> > Credit window is 1000.
>>> >
>>> > I can't find anything in my router config nor in my C client code
>>> about max
>>> > frame size. What do I get by default? Or, how can I check that?
>>> >
>>> > The way I measured throughput was that -- first -- I noticed that when
>>> I
>>> > made the test go longer, i.e. send 20 million total messages instead
>>> of the
>>> > original 1 million -- it was taking much longer than I expected. So I
>>> had
>>> > each receiver log a message every time its total received messages was
>>> > divisible by 1000.
>>> >
>>> > What I saw was that the first thousand came after 11 seconds (just
>>> about as
>>> > expected because of sender-throttle to 100/sec) but that later
>>> thousands
>>> > became slower. By the time I stopped the test -- after more than 50,000
>>> > messages per receiver -- each thousand was taking ... well ... look at
>>> this
>>> > very interesting graph that I made of one receiver's behavior.
>>> >
>>> > This graph is made by just noting the time when you receive each
>>> thousandth
>>> > message (time since test started) and graphing that -- so we expect to
>>> see
>>> > an upward-sloping straight line whose slope is determined by how long
>>> it
>>> > takes to receive each 1000 messages (should be close to 10 seconds).
>>> >
>>> > [image: messages_vs_time.jpg]
>>> >
>>> > I'm glad I graphed this! This inflection point was a total shock to me.
>>> > NOTE TO SELF: always graph everything from now on forever.
>>> >
>>> > I guess Something Interesting happened at about 28 seconds!
>>> >
>>> > Maybe what I need ... is a reading from "qdstat -m" just before and
>>> after
>>> > that inflection point !?!??
>>> >
>>> >
>>> >
>>> > On Thu, Feb 11, 2021 at 5:37 PM Ted Ross <tr...@apache.org> wrote:
>>> >
>>> > > On Thu, Feb 11, 2021 at 2:08 PM Michael Goulish <mgoulish@redhat.com
>>> >
>>> > > wrote:
>>> > >
>>> > > > OK, so in the file Dispatch Router file src/buffer.c I changed
>>> this:
>>> > > > size_t BUFFER_SIZE = 512;
>>> > > > to this:
>>> > > > size_t BUFFER_SIZE = 4096;
>>> > > >
>>> > > > Gordon tells me that's like 8 times bigger.
>>> > > >
>>> > > >
>>> > > > It makes a terrific difference in throughput in the TCP adapter,
>>> and if
>>> > > you
>>> > > > limit the sender to the throughput that the receiver can accept,
>>> it can
>>> > > go
>>> > > > Real Fast with no memory bloat. ( Like 15 Gbit/sec )
>>> > > >
>>> > > > But.
>>> > > > AMQP throughput is Not Happy with this change.
>>> > > >
>>> > > > Some of the managed fields grow rapidly (although not enough to
>>> account
>>> > > for
>>> > > > total memory growth) -- and throughput gradually drops to a crawl.
>>> > > >
>>> > > > Here are the fields that increase dramatically (like 10x or more)
>>> -- and
>>> > > > the ones that don't much change.
>>> > > >
>>> > > > qd_bitmask_t
>>> > > > *qd_buffer_t *
>>> > > > qd_composed_field_t
>>> > > > qd_composite_t
>>> > > > qd_connection_t
>>> > > > qd_hash_handle_t
>>> > > > qd_hash_item_t
>>> > > > qd_iterator_t
>>> > > > *qd_link_ref_t*
>>> > > > qd_link_t
>>> > > > qd_listener_t
>>> > > > qd_log_entry_t
>>> > > > qd_management_context_t
>>> > > > *qd_message_content_t*
>>> > > > *qd_message_t*
>>> > > > qd_node_t
>>> > > > qd_parse_node_t
>>> > > > qd_parse_tree_t
>>> > > > qd_parsed_field_t
>>> > > > qd_session_t
>>> > > > qd_timer_t
>>> > > > *qdr_action_t*
>>> > > > qdr_address_config_t
>>> > > > qdr_address_t
>>> > > > qdr_connection_info_t
>>> > > > qdr_connection_t
>>> > > > qdr_connection_work_t
>>> > > > qdr_core_timer_t
>>> > > > qdr_delivery_cleanup_t
>>> > > > *qdr_delivery_ref_t*
>>> > > > *qdr_delivery_t*
>>> > > > qdr_field_t
>>> > > > qdr_general_work_t
>>> > > > qdr_link_ref_t
>>> > > > qdr_link_t
>>> > > > qdr_link_work_t
>>> > > > qdr_query_t
>>> > > > qdr_terminus_t
>>> > > >
>>> > > >
>>> > > > Does anyone have a great idea about any experiment I could do,
>>> > > > instrumentation I could add, whatever -- that might help to further
>>> > > > diagnose what is going on?
>>> > > >
>>> > >
>>> > > Can you explain how you are measuring AMQP throughput? What message
>>> sizes
>>> > > are you using? Credit windows? How many senders and receivers?
>>> Max frame
>>> > > size?
>>> > >
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>
>>>
Re: Dispatch Router: Changing buffer size in buffer.c blows up AMQP.
Posted by Michael Goulish <mg...@redhat.com>.
A further note.
I just looked at all 100 receivers this way.
They all have the same inflection point at 28000 messages.
But ... it could also have been a particular point in time that triggered
it -- because the all hit the 28000 message mark within 2.5 seconds of each
other.
On Fri, Feb 12, 2021 at 11:14 AM Michael Goulish <mg...@redhat.com>
wrote:
> Sorry -- I didn't realize this list would remove the image of my graph.
>
> Can everyone see this?
> <https://www.dropbox.com/s/4t1xbp46y57mfgn/messages_vs_time.jpg?dl=0>
>
> On Fri, Feb 12, 2021 at 7:51 AM Chuck Rolke <cr...@redhat.com> wrote:
>
>> The mail list scrubs attachments. Maybe create a jira and add the image
>> to that.
>>
>> ----- Original Message -----
>> > From: "Michael Goulish" <mg...@redhat.com>
>> > To: users@qpid.apache.org
>> > Sent: Friday, February 12, 2021 2:43:40 AM
>> > Subject: Re: Dispatch Router: Changing buffer size in buffer.c blows up
>> AMQP.
>> >
>> > *Can you explain how you are measuring AMQP throughput? What message
>> sizes
>> > are you using? Credit windows? How many senders and receivers? Max
>> frame*
>> > * size?*
>> >
>> > Oops! Good point. Describe the Test!
>> >
>> > 100 senders, 100 receivers, 100 unique addresses -- each sender sends to
>> > one receiver.
>> > Each sender is throttled to 100 messages per second (Apparently I Really
>> > Like the number 100).
>> > And message size is .... wait for it ... 100. (payload size .. so
>> > really 139 or something like that.)
>> >
>> > Credit window is 1000.
>> >
>> > I can't find anything in my router config nor in my C client code about
>> max
>> > frame size. What do I get by default? Or, how can I check that?
>> >
>> > The way I measured throughput was that -- first -- I noticed that when I
>> > made the test go longer, i.e. send 20 million total messages instead of
>> the
>> > original 1 million -- it was taking much longer than I expected. So I
>> had
>> > each receiver log a message every time its total received messages was
>> > divisible by 1000.
>> >
>> > What I saw was that the first thousand came after 11 seconds (just
>> about as
>> > expected because of sender-throttle to 100/sec) but that later thousands
>> > became slower. By the time I stopped the test -- after more than 50,000
>> > messages per receiver -- each thousand was taking ... well ... look at
>> this
>> > very interesting graph that I made of one receiver's behavior.
>> >
>> > This graph is made by just noting the time when you receive each
>> thousandth
>> > message (time since test started) and graphing that -- so we expect to
>> see
>> > an upward-sloping straight line whose slope is determined by how long it
>> > takes to receive each 1000 messages (should be close to 10 seconds).
>> >
>> > [image: messages_vs_time.jpg]
>> >
>> > I'm glad I graphed this! This inflection point was a total shock to me.
>> > NOTE TO SELF: always graph everything from now on forever.
>> >
>> > I guess Something Interesting happened at about 28 seconds!
>> >
>> > Maybe what I need ... is a reading from "qdstat -m" just before and
>> after
>> > that inflection point !?!??
>> >
>> >
>> >
>> > On Thu, Feb 11, 2021 at 5:37 PM Ted Ross <tr...@apache.org> wrote:
>> >
>> > > On Thu, Feb 11, 2021 at 2:08 PM Michael Goulish <mg...@redhat.com>
>> > > wrote:
>> > >
>> > > > OK, so in the file Dispatch Router file src/buffer.c I changed this:
>> > > > size_t BUFFER_SIZE = 512;
>> > > > to this:
>> > > > size_t BUFFER_SIZE = 4096;
>> > > >
>> > > > Gordon tells me that's like 8 times bigger.
>> > > >
>> > > >
>> > > > It makes a terrific difference in throughput in the TCP adapter,
>> and if
>> > > you
>> > > > limit the sender to the throughput that the receiver can accept, it
>> can
>> > > go
>> > > > Real Fast with no memory bloat. ( Like 15 Gbit/sec )
>> > > >
>> > > > But.
>> > > > AMQP throughput is Not Happy with this change.
>> > > >
>> > > > Some of the managed fields grow rapidly (although not enough to
>> account
>> > > for
>> > > > total memory growth) -- and throughput gradually drops to a crawl.
>> > > >
>> > > > Here are the fields that increase dramatically (like 10x or more)
>> -- and
>> > > > the ones that don't much change.
>> > > >
>> > > > qd_bitmask_t
>> > > > *qd_buffer_t *
>> > > > qd_composed_field_t
>> > > > qd_composite_t
>> > > > qd_connection_t
>> > > > qd_hash_handle_t
>> > > > qd_hash_item_t
>> > > > qd_iterator_t
>> > > > *qd_link_ref_t*
>> > > > qd_link_t
>> > > > qd_listener_t
>> > > > qd_log_entry_t
>> > > > qd_management_context_t
>> > > > *qd_message_content_t*
>> > > > *qd_message_t*
>> > > > qd_node_t
>> > > > qd_parse_node_t
>> > > > qd_parse_tree_t
>> > > > qd_parsed_field_t
>> > > > qd_session_t
>> > > > qd_timer_t
>> > > > *qdr_action_t*
>> > > > qdr_address_config_t
>> > > > qdr_address_t
>> > > > qdr_connection_info_t
>> > > > qdr_connection_t
>> > > > qdr_connection_work_t
>> > > > qdr_core_timer_t
>> > > > qdr_delivery_cleanup_t
>> > > > *qdr_delivery_ref_t*
>> > > > *qdr_delivery_t*
>> > > > qdr_field_t
>> > > > qdr_general_work_t
>> > > > qdr_link_ref_t
>> > > > qdr_link_t
>> > > > qdr_link_work_t
>> > > > qdr_query_t
>> > > > qdr_terminus_t
>> > > >
>> > > >
>> > > > Does anyone have a great idea about any experiment I could do,
>> > > > instrumentation I could add, whatever -- that might help to further
>> > > > diagnose what is going on?
>> > > >
>> > >
>> > > Can you explain how you are measuring AMQP throughput? What message
>> sizes
>> > > are you using? Credit windows? How many senders and receivers? Max
>> frame
>> > > size?
>> > >
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
Re: Dispatch Router: Changing buffer size in buffer.c blows up AMQP.
Posted by Michael Goulish <mg...@redhat.com>.
Sorry -- I didn't realize this list would remove the image of my graph.
Can everyone see this?
<https://www.dropbox.com/s/4t1xbp46y57mfgn/messages_vs_time.jpg?dl=0>
On Fri, Feb 12, 2021 at 7:51 AM Chuck Rolke <cr...@redhat.com> wrote:
> The mail list scrubs attachments. Maybe create a jira and add the image
> to that.
>
> ----- Original Message -----
> > From: "Michael Goulish" <mg...@redhat.com>
> > To: users@qpid.apache.org
> > Sent: Friday, February 12, 2021 2:43:40 AM
> > Subject: Re: Dispatch Router: Changing buffer size in buffer.c blows up
> AMQP.
> >
> > *Can you explain how you are measuring AMQP throughput? What message
> sizes
> > are you using? Credit windows? How many senders and receivers? Max
> frame*
> > * size?*
> >
> > Oops! Good point. Describe the Test!
> >
> > 100 senders, 100 receivers, 100 unique addresses -- each sender sends to
> > one receiver.
> > Each sender is throttled to 100 messages per second (Apparently I Really
> > Like the number 100).
> > And message size is .... wait for it ... 100. (payload size .. so
> > really 139 or something like that.)
> >
> > Credit window is 1000.
> >
> > I can't find anything in my router config nor in my C client code about
> max
> > frame size. What do I get by default? Or, how can I check that?
> >
> > The way I measured throughput was that -- first -- I noticed that when I
> > made the test go longer, i.e. send 20 million total messages instead of
> the
> > original 1 million -- it was taking much longer than I expected. So I had
> > each receiver log a message every time its total received messages was
> > divisible by 1000.
> >
> > What I saw was that the first thousand came after 11 seconds (just about
> as
> > expected because of sender-throttle to 100/sec) but that later thousands
> > became slower. By the time I stopped the test -- after more than 50,000
> > messages per receiver -- each thousand was taking ... well ... look at
> this
> > very interesting graph that I made of one receiver's behavior.
> >
> > This graph is made by just noting the time when you receive each
> thousandth
> > message (time since test started) and graphing that -- so we expect to
> see
> > an upward-sloping straight line whose slope is determined by how long it
> > takes to receive each 1000 messages (should be close to 10 seconds).
> >
> > [image: messages_vs_time.jpg]
> >
> > I'm glad I graphed this! This inflection point was a total shock to me.
> > NOTE TO SELF: always graph everything from now on forever.
> >
> > I guess Something Interesting happened at about 28 seconds!
> >
> > Maybe what I need ... is a reading from "qdstat -m" just before and after
> > that inflection point !?!??
> >
> >
> >
> > On Thu, Feb 11, 2021 at 5:37 PM Ted Ross <tr...@apache.org> wrote:
> >
> > > On Thu, Feb 11, 2021 at 2:08 PM Michael Goulish <mg...@redhat.com>
> > > wrote:
> > >
> > > > OK, so in the file Dispatch Router file src/buffer.c I changed this:
> > > > size_t BUFFER_SIZE = 512;
> > > > to this:
> > > > size_t BUFFER_SIZE = 4096;
> > > >
> > > > Gordon tells me that's like 8 times bigger.
> > > >
> > > >
> > > > It makes a terrific difference in throughput in the TCP adapter, and
> if
> > > you
> > > > limit the sender to the throughput that the receiver can accept, it
> can
> > > go
> > > > Real Fast with no memory bloat. ( Like 15 Gbit/sec )
> > > >
> > > > But.
> > > > AMQP throughput is Not Happy with this change.
> > > >
> > > > Some of the managed fields grow rapidly (although not enough to
> account
> > > for
> > > > total memory growth) -- and throughput gradually drops to a crawl.
> > > >
> > > > Here are the fields that increase dramatically (like 10x or more) --
> and
> > > > the ones that don't much change.
> > > >
> > > > qd_bitmask_t
> > > > *qd_buffer_t *
> > > > qd_composed_field_t
> > > > qd_composite_t
> > > > qd_connection_t
> > > > qd_hash_handle_t
> > > > qd_hash_item_t
> > > > qd_iterator_t
> > > > *qd_link_ref_t*
> > > > qd_link_t
> > > > qd_listener_t
> > > > qd_log_entry_t
> > > > qd_management_context_t
> > > > *qd_message_content_t*
> > > > *qd_message_t*
> > > > qd_node_t
> > > > qd_parse_node_t
> > > > qd_parse_tree_t
> > > > qd_parsed_field_t
> > > > qd_session_t
> > > > qd_timer_t
> > > > *qdr_action_t*
> > > > qdr_address_config_t
> > > > qdr_address_t
> > > > qdr_connection_info_t
> > > > qdr_connection_t
> > > > qdr_connection_work_t
> > > > qdr_core_timer_t
> > > > qdr_delivery_cleanup_t
> > > > *qdr_delivery_ref_t*
> > > > *qdr_delivery_t*
> > > > qdr_field_t
> > > > qdr_general_work_t
> > > > qdr_link_ref_t
> > > > qdr_link_t
> > > > qdr_link_work_t
> > > > qdr_query_t
> > > > qdr_terminus_t
> > > >
> > > >
> > > > Does anyone have a great idea about any experiment I could do,
> > > > instrumentation I could add, whatever -- that might help to further
> > > > diagnose what is going on?
> > > >
> > >
> > > Can you explain how you are measuring AMQP throughput? What message
> sizes
> > > are you using? Credit windows? How many senders and receivers? Max
> frame
> > > size?
> > >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Dispatch Router: Changing buffer size in buffer.c blows up
AMQP.
Posted by Chuck Rolke <cr...@redhat.com>.
The mail list scrubs attachments. Maybe create a jira and add the image
to that.
----- Original Message -----
> From: "Michael Goulish" <mg...@redhat.com>
> To: users@qpid.apache.org
> Sent: Friday, February 12, 2021 2:43:40 AM
> Subject: Re: Dispatch Router: Changing buffer size in buffer.c blows up AMQP.
>
> *Can you explain how you are measuring AMQP throughput? What message sizes
> are you using? Credit windows? How many senders and receivers? Max frame*
> * size?*
>
> Oops! Good point. Describe the Test!
>
> 100 senders, 100 receivers, 100 unique addresses -- each sender sends to
> one receiver.
> Each sender is throttled to 100 messages per second (Apparently I Really
> Like the number 100).
> And message size is .... wait for it ... 100. (payload size .. so
> really 139 or something like that.)
>
> Credit window is 1000.
>
> I can't find anything in my router config nor in my C client code about max
> frame size. What do I get by default? Or, how can I check that?
>
> The way I measured throughput was that -- first -- I noticed that when I
> made the test go longer, i.e. send 20 million total messages instead of the
> original 1 million -- it was taking much longer than I expected. So I had
> each receiver log a message every time its total received messages was
> divisible by 1000.
>
> What I saw was that the first thousand came after 11 seconds (just about as
> expected because of sender-throttle to 100/sec) but that later thousands
> became slower. By the time I stopped the test -- after more than 50,000
> messages per receiver -- each thousand was taking ... well ... look at this
> very interesting graph that I made of one receiver's behavior.
>
> This graph is made by just noting the time when you receive each thousandth
> message (time since test started) and graphing that -- so we expect to see
> an upward-sloping straight line whose slope is determined by how long it
> takes to receive each 1000 messages (should be close to 10 seconds).
>
> [image: messages_vs_time.jpg]
>
> I'm glad I graphed this! This inflection point was a total shock to me.
> NOTE TO SELF: always graph everything from now on forever.
>
> I guess Something Interesting happened at about 28 seconds!
>
> Maybe what I need ... is a reading from "qdstat -m" just before and after
> that inflection point !?!??
>
>
>
> On Thu, Feb 11, 2021 at 5:37 PM Ted Ross <tr...@apache.org> wrote:
>
> > On Thu, Feb 11, 2021 at 2:08 PM Michael Goulish <mg...@redhat.com>
> > wrote:
> >
> > > OK, so in the file Dispatch Router file src/buffer.c I changed this:
> > > size_t BUFFER_SIZE = 512;
> > > to this:
> > > size_t BUFFER_SIZE = 4096;
> > >
> > > Gordon tells me that's like 8 times bigger.
> > >
> > >
> > > It makes a terrific difference in throughput in the TCP adapter, and if
> > you
> > > limit the sender to the throughput that the receiver can accept, it can
> > go
> > > Real Fast with no memory bloat. ( Like 15 Gbit/sec )
> > >
> > > But.
> > > AMQP throughput is Not Happy with this change.
> > >
> > > Some of the managed fields grow rapidly (although not enough to account
> > for
> > > total memory growth) -- and throughput gradually drops to a crawl.
> > >
> > > Here are the fields that increase dramatically (like 10x or more) -- and
> > > the ones that don't much change.
> > >
> > > qd_bitmask_t
> > > *qd_buffer_t *
> > > qd_composed_field_t
> > > qd_composite_t
> > > qd_connection_t
> > > qd_hash_handle_t
> > > qd_hash_item_t
> > > qd_iterator_t
> > > *qd_link_ref_t*
> > > qd_link_t
> > > qd_listener_t
> > > qd_log_entry_t
> > > qd_management_context_t
> > > *qd_message_content_t*
> > > *qd_message_t*
> > > qd_node_t
> > > qd_parse_node_t
> > > qd_parse_tree_t
> > > qd_parsed_field_t
> > > qd_session_t
> > > qd_timer_t
> > > *qdr_action_t*
> > > qdr_address_config_t
> > > qdr_address_t
> > > qdr_connection_info_t
> > > qdr_connection_t
> > > qdr_connection_work_t
> > > qdr_core_timer_t
> > > qdr_delivery_cleanup_t
> > > *qdr_delivery_ref_t*
> > > *qdr_delivery_t*
> > > qdr_field_t
> > > qdr_general_work_t
> > > qdr_link_ref_t
> > > qdr_link_t
> > > qdr_link_work_t
> > > qdr_query_t
> > > qdr_terminus_t
> > >
> > >
> > > Does anyone have a great idea about any experiment I could do,
> > > instrumentation I could add, whatever -- that might help to further
> > > diagnose what is going on?
> > >
> >
> > Can you explain how you are measuring AMQP throughput? What message sizes
> > are you using? Credit windows? How many senders and receivers? Max frame
> > size?
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Dispatch Router: Changing buffer size in buffer.c blows up AMQP.
Posted by Michael Goulish <mg...@redhat.com>.
*Can you explain how you are measuring AMQP throughput? What message sizes
are you using? Credit windows? How many senders and receivers? Max frame*
* size?*
Oops! Good point. Describe the Test!
100 senders, 100 receivers, 100 unique addresses -- each sender sends to
one receiver.
Each sender is throttled to 100 messages per second (Apparently I Really
Like the number 100).
And message size is .... wait for it ... 100. (payload size .. so
really 139 or something like that.)
Credit window is 1000.
I can't find anything in my router config nor in my C client code about max
frame size. What do I get by default? Or, how can I check that?
The way I measured throughput was that -- first -- I noticed that when I
made the test go longer, i.e. send 20 million total messages instead of the
original 1 million -- it was taking much longer than I expected. So I had
each receiver log a message every time its total received messages was
divisible by 1000.
What I saw was that the first thousand came after 11 seconds (just about as
expected because of sender-throttle to 100/sec) but that later thousands
became slower. By the time I stopped the test -- after more than 50,000
messages per receiver -- each thousand was taking ... well ... look at this
very interesting graph that I made of one receiver's behavior.
This graph is made by just noting the time when you receive each thousandth
message (time since test started) and graphing that -- so we expect to see
an upward-sloping straight line whose slope is determined by how long it
takes to receive each 1000 messages (should be close to 10 seconds).
[image: messages_vs_time.jpg]
I'm glad I graphed this! This inflection point was a total shock to me.
NOTE TO SELF: always graph everything from now on forever.
I guess Something Interesting happened at about 28 seconds!
Maybe what I need ... is a reading from "qdstat -m" just before and after
that inflection point !?!??
On Thu, Feb 11, 2021 at 5:37 PM Ted Ross <tr...@apache.org> wrote:
> On Thu, Feb 11, 2021 at 2:08 PM Michael Goulish <mg...@redhat.com>
> wrote:
>
> > OK, so in the file Dispatch Router file src/buffer.c I changed this:
> > size_t BUFFER_SIZE = 512;
> > to this:
> > size_t BUFFER_SIZE = 4096;
> >
> > Gordon tells me that's like 8 times bigger.
> >
> >
> > It makes a terrific difference in throughput in the TCP adapter, and if
> you
> > limit the sender to the throughput that the receiver can accept, it can
> go
> > Real Fast with no memory bloat. ( Like 15 Gbit/sec )
> >
> > But.
> > AMQP throughput is Not Happy with this change.
> >
> > Some of the managed fields grow rapidly (although not enough to account
> for
> > total memory growth) -- and throughput gradually drops to a crawl.
> >
> > Here are the fields that increase dramatically (like 10x or more) -- and
> > the ones that don't much change.
> >
> > qd_bitmask_t
> > *qd_buffer_t *
> > qd_composed_field_t
> > qd_composite_t
> > qd_connection_t
> > qd_hash_handle_t
> > qd_hash_item_t
> > qd_iterator_t
> > *qd_link_ref_t*
> > qd_link_t
> > qd_listener_t
> > qd_log_entry_t
> > qd_management_context_t
> > *qd_message_content_t*
> > *qd_message_t*
> > qd_node_t
> > qd_parse_node_t
> > qd_parse_tree_t
> > qd_parsed_field_t
> > qd_session_t
> > qd_timer_t
> > *qdr_action_t*
> > qdr_address_config_t
> > qdr_address_t
> > qdr_connection_info_t
> > qdr_connection_t
> > qdr_connection_work_t
> > qdr_core_timer_t
> > qdr_delivery_cleanup_t
> > *qdr_delivery_ref_t*
> > *qdr_delivery_t*
> > qdr_field_t
> > qdr_general_work_t
> > qdr_link_ref_t
> > qdr_link_t
> > qdr_link_work_t
> > qdr_query_t
> > qdr_terminus_t
> >
> >
> > Does anyone have a great idea about any experiment I could do,
> > instrumentation I could add, whatever -- that might help to further
> > diagnose what is going on?
> >
>
> Can you explain how you are measuring AMQP throughput? What message sizes
> are you using? Credit windows? How many senders and receivers? Max frame
> size?
>
Re: Dispatch Router: Changing buffer size in buffer.c blows up AMQP.
Posted by Ted Ross <tr...@apache.org>.
On Thu, Feb 11, 2021 at 2:08 PM Michael Goulish <mg...@redhat.com> wrote:
> OK, so in the file Dispatch Router file src/buffer.c I changed this:
> size_t BUFFER_SIZE = 512;
> to this:
> size_t BUFFER_SIZE = 4096;
>
> Gordon tells me that's like 8 times bigger.
>
>
> It makes a terrific difference in throughput in the TCP adapter, and if you
> limit the sender to the throughput that the receiver can accept, it can go
> Real Fast with no memory bloat. ( Like 15 Gbit/sec )
>
> But.
> AMQP throughput is Not Happy with this change.
>
> Some of the managed fields grow rapidly (although not enough to account for
> total memory growth) -- and throughput gradually drops to a crawl.
>
> Here are the fields that increase dramatically (like 10x or more) -- and
> the ones that don't much change.
>
> qd_bitmask_t
> *qd_buffer_t *
> qd_composed_field_t
> qd_composite_t
> qd_connection_t
> qd_hash_handle_t
> qd_hash_item_t
> qd_iterator_t
> *qd_link_ref_t*
> qd_link_t
> qd_listener_t
> qd_log_entry_t
> qd_management_context_t
> *qd_message_content_t*
> *qd_message_t*
> qd_node_t
> qd_parse_node_t
> qd_parse_tree_t
> qd_parsed_field_t
> qd_session_t
> qd_timer_t
> *qdr_action_t*
> qdr_address_config_t
> qdr_address_t
> qdr_connection_info_t
> qdr_connection_t
> qdr_connection_work_t
> qdr_core_timer_t
> qdr_delivery_cleanup_t
> *qdr_delivery_ref_t*
> *qdr_delivery_t*
> qdr_field_t
> qdr_general_work_t
> qdr_link_ref_t
> qdr_link_t
> qdr_link_work_t
> qdr_query_t
> qdr_terminus_t
>
>
> Does anyone have a great idea about any experiment I could do,
> instrumentation I could add, whatever -- that might help to further
> diagnose what is going on?
>
Can you explain how you are measuring AMQP throughput? What message sizes
are you using? Credit windows? How many senders and receivers? Max frame
size?