You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Michael Goulish <mg...@redhat.com> on 2021/03/15 09:37:22 UTC
Dispatch Router TCP Latency study...
...over a 40 Gbit/sec link, using the Golang 'Hey' HTTP load generator, and
a simple Go webserver, with and without the Dispatch Router in the middle.
Document here
<https://drive.google.com/file/d/1YJXuDqDWHDLTdCglDeK6YZqx6xAV_NII/view?usp=sharing>,
because Apache won't let me send images in emails.
Please give suggestions on this thread about changes you would like to see
in the test or the measurements captured.
Re: Dispatch Router TCP Latency study...
Posted by Michael Goulish <mg...@redhat.com>.
Oops. I should say -- these tests were done with Dispatch Router
src/buffer.c BUFFER_SIZE = 4096.
Should probably put that in the title...
On Mon, Mar 15, 2021 at 5:37 AM Michael Goulish <mg...@redhat.com> wrote:
> ...over a 40 Gbit/sec link, using the Golang 'Hey' HTTP load generator,
> and a simple Go webserver, with and without the Dispatch Router in the
> middle.
>
> Document here
> <https://drive.google.com/file/d/1YJXuDqDWHDLTdCglDeK6YZqx6xAV_NII/view?usp=sharing>,
> because Apache won't let me send images in emails.
>
> Please give suggestions on this thread about changes you would like to see
> in the test or the measurements captured.
>
>
Re: Dispatch Router TCP Latency study...
Posted by Gordon Sim <gs...@redhat.com>.
On 15/03/2021 16:12, Michael Goulish wrote:
> *So why is the request/second rate significantly lower with the router? *
>
> My model, so far, is as follows. Does this seem plausible?
>
> When there is no router present the senders send with less constraint,
> filling up whatever buffers there are between them and the server. This
> achieves high throughput but at the cost of higher latency. If they could
> somehow restrain themselves they would probably achieve significantly lower
> latency than the with-router case.
>
> When the router is present it imposes more constraints on how fast the
> senders can send. This reduces throughput but improves latency to the point
> where the with-router case is actually slightly faster along much of the
> curve, although at higher levels of concurrency the difference is very
> small.
If latency is measured from the point that the request has been fully
written to the socket on the client, up until the point that the
response has been read from it, then in theory I agree that the rate
could be lower without the latency being any worse. Any time waiting for
the socket to be writable would not be included in the latency but would
affect the rate.
I would expect latency to be measured from the time the request was
*triggered* until the response was received. In which case any time
waiting for the socket to be writable *would* impact the latency.
For the client socket not to be writable the local buffers would be full
and the router would have stopped reading from that socket on the other
side. I wonder if some instrumentation to show the time on the router
where the socket was not being read from would help confirm the issue?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Dispatch Router TCP Latency study...
Posted by Michael Goulish <mg...@redhat.com>.
*So why is the request/second rate significantly lower with the router? *
My model, so far, is as follows. Does this seem plausible?
When there is no router present the senders send with less constraint,
filling up whatever buffers there are between them and the server. This
achieves high throughput but at the cost of higher latency. If they could
somehow restrain themselves they would probably achieve significantly lower
latency than the with-router case.
When the router is present it imposes more constraints on how fast the
senders can send. This reduces throughput but improves latency to the point
where the with-router case is actually slightly faster along much of the
curve, although at higher levels of concurrency the difference is very
small.
On Mon, Mar 15, 2021 at 12:06 PM Michael Goulish <mg...@redhat.com>
wrote:
> I have updated the document to include "4 KB Buffers" in title, and new
> graphs for Average Latency, and Slowest Latency.
>
> The link I already sent still works for the new version.
>
>
> On Mon, Mar 15, 2021 at 10:51 AM Gordon Sim <gs...@redhat.com> wrote:
>
>> On 15/03/2021 14:06, Michael Goulish wrote:
>> > *How does the average latency compare?*
>> >
>> > I had to check this 3 times before I believed it, but the average
>> latency
>> > is dead equal -- to within 0.1 msec -- with or without the router, all
>> the
>> > way from 10 to 100 concurrent workers.
>>
>> So why is the request/second rate significantly lower with the router?
>> If each client is just issuing one request after the other, and the
>> average time to finish each request is as good with the router as
>> without, where is the extra time that a reduced rate implies going?
>>
>> Could it be that the small number of slower requests are so much slower
>> that it is impacting the overall rate?
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
Re: Dispatch Router TCP Latency study...
Posted by Michael Goulish <mg...@redhat.com>.
I have updated the document to include "4 KB Buffers" in title, and new
graphs for Average Latency, and Slowest Latency.
The link I already sent still works for the new version.
On Mon, Mar 15, 2021 at 10:51 AM Gordon Sim <gs...@redhat.com> wrote:
> On 15/03/2021 14:06, Michael Goulish wrote:
> > *How does the average latency compare?*
> >
> > I had to check this 3 times before I believed it, but the average latency
> > is dead equal -- to within 0.1 msec -- with or without the router, all
> the
> > way from 10 to 100 concurrent workers.
>
> So why is the request/second rate significantly lower with the router?
> If each client is just issuing one request after the other, and the
> average time to finish each request is as good with the router as
> without, where is the extra time that a reduced rate implies going?
>
> Could it be that the small number of slower requests are so much slower
> that it is impacting the overall rate?
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Dispatch Router TCP Latency study...
Posted by Gordon Sim <gs...@redhat.com>.
On 15/03/2021 14:06, Michael Goulish wrote:
> *How does the average latency compare?*
>
> I had to check this 3 times before I believed it, but the average latency
> is dead equal -- to within 0.1 msec -- with or without the router, all the
> way from 10 to 100 concurrent workers.
So why is the request/second rate significantly lower with the router?
If each client is just issuing one request after the other, and the
average time to finish each request is as good with the router as
without, where is the extra time that a reduced rate implies going?
Could it be that the small number of slower requests are so much slower
that it is impacting the overall rate?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Dispatch Router TCP Latency study...
Posted by Michael Goulish <mg...@redhat.com>.
*How does the average latency compare?*
I had to check this 3 times before I believed it, but the average latency
is dead equal -- to within 0.1 msec -- with or without the router, all the
way from 10 to 100 concurrent workers.
Ah, so what that means is that although 99% of responses are a tiny bit
quicker with the router, the slowest ones manage to compensate for that.
I am adding a new graph for average latency to the paper, and I will also
add a graph for 100% of latencies, i.e. "what was the slowest?" I guess
we will see there that slowest case with the router is always somewhat
worse than slowest case without. Which makes sense...
On Mon, Mar 15, 2021 at 6:17 AM Gordon Sim <gs...@redhat.com> wrote:
> On 15/03/2021 09:37, Michael Goulish wrote:
> > ...over a 40 Gbit/sec link, using the Golang 'Hey' HTTP load generator,
> and
> > a simple Go webserver, with and without the Dispatch Router in the
> middle.
> >
> > Document here
> > <
> https://drive.google.com/file/d/1YJXuDqDWHDLTdCglDeK6YZqx6xAV_NII/view?usp=sharing
> >,
> > because Apache won't let me send images in emails.
> >
> > Please give suggestions on this thread about changes you would like to
> see
> > in the test or the measurements captured.
> >
>
> How does the average latency compare?
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Dispatch Router TCP Latency study...
Posted by Gordon Sim <gs...@redhat.com>.
On 15/03/2021 09:37, Michael Goulish wrote:
> ...over a 40 Gbit/sec link, using the Golang 'Hey' HTTP load generator, and
> a simple Go webserver, with and without the Dispatch Router in the middle.
>
> Document here
> <https://drive.google.com/file/d/1YJXuDqDWHDLTdCglDeK6YZqx6xAV_NII/view?usp=sharing>,
> because Apache won't let me send images in emails.
>
> Please give suggestions on this thread about changes you would like to see
> in the test or the measurements captured.
>
How does the average latency compare?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org