You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by tomt <tt...@netapp.com> on 2020/07/09 14:43:48 UTC
default limits of qpid-c++ broker, dispatch router, or proton?
Hello,
I have been trying to determine the highest rate of messages that can be
sent at any given time through my environment that includes the qpid-cpp
broker and the dispatch router all using AMQP 1.0 (via Proton).
There seems to be some maximum cap that only lets me send up to 2500
messages/sec and I'm trying to determine if that's a limitation of my
environment or perhaps a default in any of the mentioned packages. I have 2
senders blasting messages and 10 receivers subscribing to all of them.
I have searched around and it looks like any sort of limiting is off by
default in the broker and router, but that 2500 number is consistent and
makes me think that there could be a limiter somewhere. Any sort of
pointers or suggestions would be greatly appreciated.
Thank you.
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: default limits of qpid-c++ broker, dispatch router, or proton?
Posted by tomt <tt...@netapp.com>.
Thanks Chuck!
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: default limits of qpid-c++ broker, dispatch router, or proton?
Posted by Chuck Rolke <cr...@redhat.com>.
qpid-dispatch includes some tooling to help with AMQP operations in the router.
https://github.com/apache/qpid-dispatch/tree/master/tools/scraper
To get started use a single router with appropriate logging turned on.
Then run your tests to generate logs:
only a few hundred messages
short unique transfer payloads
Run scraper on log file(s) to generate a web page to be viewed with browser.
You can then see credit starvation, if any, in the Connections section.
Individual message transfers through the router and related settlements are
presented multiple ways.
There's a lot there; check it out.
-Chuck
----- Original Message -----
> From: "tomt" <tt...@netapp.com>
> To: users@qpid.apache.org
> Sent: Friday, July 10, 2020 3:49:19 PM
> Subject: Re: default limits of qpid-c++ broker, dispatch router, or proton?
>
> You're right about the Python client. We have a websocket client utility
> that connects to the apache webserver and then locally we have a port that
> opens that we connect the python client to that.
>
> I'll try the different combinations. Thanks
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: default limits of qpid-c++ broker, dispatch router, or proton?
Posted by tomt <tt...@netapp.com>.
You're right about the Python client. We have a websocket client utility
that connects to the apache webserver and then locally we have a port that
opens that we connect the python client to that.
I'll try the different combinations. Thanks
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: default limits of qpid-c++ broker, dispatch router, or proton?
Posted by Gordon Sim <gs...@redhat.com>.
On 10/07/2020 7:51 pm, tomt wrote:
> MY configuration is that I have my receivers connecting to the router
> (through an Apache web server) via websockets. The topics they subscribe to
> live in the broker.
>
> I have link routing configured for this, so I am getting real flow control
> (based on what I read in the link above)
The python client doesn't support websockets, does it? My suggestion
would be to do some comparative tests using the router only and the
broker only and see if that shows anything different.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: default limits of qpid-c++ broker, dispatch router, or proton?
Posted by tomt <tt...@netapp.com>.
MY configuration is that I have my receivers connecting to the router
(through an Apache web server) via websockets. The topics they subscribe to
live in the broker.
I have link routing configured for this, so I am getting real flow control
(based on what I read in the link above)
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: default limits of qpid-c++ broker, dispatch router, or proton?
Posted by Gordon Sim <gs...@redhat.com>.
On 10/07/2020 4:58 pm, Robbie Gemmell wrote:
> When you have the router in between the broker and producer/consumer
> clients, then what happens will depend on how things have been set up
> to operate
I had assumed you were testing with the broker *or* the router, but
re-reading after this I see I am likely wrong there. Are you using them
together? If so, I would certainly try running against them individually
for comparison.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: default limits of qpid-c++ broker, dispatch router, or proton?
Posted by Robbie Gemmell <ro...@gmail.com>.
The behaviour is going to be dependent on how your components are
actually configured, so it's hard to say specifics without more detail
on how e.g your router and broker are actually configured to work
together with the clients.
Senders have no ability to control credit, it is granted by the
receiver end of a link such that the sending end of the link can
transfer messages across to it. For example in a simple client-broker
usage, the broker will grant credit to a sender allow messages to be
produced onto its queues/topics, and a consuming client will grant
credit to the broker to receive messages from its queue/topic, but the
two message flows are independent, the sender client doesn't see
credit from the receiver client, they each transfer messages
independently to/from the broker.
When you have the router in between the broker and producer/consumer
clients, then what happens will depend on how things have been set up
to operate, e.g whether you are using routing links through to the
broker, or if the router is message-routing messages to the broker.
See http://qpid.apache.org/releases/qpid-dispatch-1.12.0/user-guide/index.html#how-routers-route-messages-qdr
for an overview of these. That will influence where credit is
effectively being granted to/from and how messages flow between
components. E.g with message routing the router itself is then
granting credit to senders, and itself being granted credit from the
broker, but it doesn't take responsibility for accepting/other the
messages, that would still happen end to end between the client and
broker and vice versa.
Robbie
On Fri, 10 Jul 2020 at 16:03, tomt <tt...@netapp.com> wrote:
>
> Thanks for the fast response. I spent a good part of the afternoon looking
> into the whole flow control system to understand better given what you had
> asked.
>
> I am using the Python client as my receivers and the C++ API as my senders
> who each synchronously send on their own threads as fast as possible.
>
> I dug a little deeper and found the Python client starts off receivers with
> 10 credits as the default since I have not taken credits into account for
> anything I've done so far. I am having an assumption here that increasing
> the number will increase my throughput, but I think I still lack some
> general understanding here.
>
> If I have multiple receivers subscribed to a given topic and they each have
> some X amount of credits, how does the sender account for this when sending?
> Some receivers could be slower in replenishing credits than others and the
> sender only sends to the topic and has no idea how many receivers there are.
> After further looking at the examples and AMQP 1.0 spec, it doesn't seem
> like I have much ability at all to control credits from the sender side at
> all. Do you have anything I can look at to learn more about the flow
> control feature?
>
> Thanks
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: default limits of qpid-c++ broker, dispatch router, or proton?
Posted by Robbie Gemmell <ro...@gmail.com>.
On Fri, 10 Jul 2020 at 16:54, Gordon Sim <gs...@redhat.com> wrote:
>
> On 10/07/2020 4:03 pm, tomt wrote:
> > Thanks for the fast response. I spent a good part of the afternoon looking
> > into the whole flow control system to understand better given what you had
> > asked.
> >
> > I am using the Python client as my receivers and the C++ API as my senders
> > who each synchronously send on their own threads as fast as possible.
>
> Which c++ API? proton? And when you say 'synchronously' what do you mean
> exactly? Are you waiting for the acknowledgement of one message before
> sending the next?
>
> The python client is by far the slowest, though I am still surprised by
> your numbers there. I would expect better even from python, especially
> if there are more than one of them (or are they each getting all the
> messages?).
>
> > I dug a little deeper and found the Python client starts off receivers with
> > 10 credits as the default since I have not taken credits into account for
> > anything I've done so far. I am having an assumption here that increasing
> > the number will increase my throughput, but I think I still lack some
> > general understanding here.
>
> A window of 10 is not terrible but I would expect you to get better
> numbers if you increase it to 100 or 250 (probably won't make much
> difference above that).
>
> >
> > If I have multiple receivers subscribed to a given topic and they each have
> > some X amount of credits, how does the sender account for this when sending?
> > Some receivers could be slower in replenishing credits than others and the
> > sender only sends to the topic and has no idea how many receivers there are.
> > After further looking at the examples and AMQP 1.0 spec, it doesn't seem
> > like I have much ability at all to control credits from the sender side at
> > all. Do you have anything I can look at to learn more about the flow
> > control feature?
>
> Credit is always issued by the process receiving messages. So for the
> senders, it is the router or the broker that will grant them credit. The
> broker and the router behave in quite different ways, but as you are
> seeing the same low rate in each case I don't think it is the server
> side that is the issue.
>
I read the original message as being not an either/or case of broker
and router, but the setup always having both? Again points to some
more detail on how things are configured being useful.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: default limits of qpid-c++ broker, dispatch router, or proton?
Posted by Gordon Sim <gs...@redhat.com>.
On 10/07/2020 4:03 pm, tomt wrote:
> Thanks for the fast response. I spent a good part of the afternoon looking
> into the whole flow control system to understand better given what you had
> asked.
>
> I am using the Python client as my receivers and the C++ API as my senders
> who each synchronously send on their own threads as fast as possible.
Which c++ API? proton? And when you say 'synchronously' what do you mean
exactly? Are you waiting for the acknowledgement of one message before
sending the next?
The python client is by far the slowest, though I am still surprised by
your numbers there. I would expect better even from python, especially
if there are more than one of them (or are they each getting all the
messages?).
> I dug a little deeper and found the Python client starts off receivers with
> 10 credits as the default since I have not taken credits into account for
> anything I've done so far. I am having an assumption here that increasing
> the number will increase my throughput, but I think I still lack some
> general understanding here.
A window of 10 is not terrible but I would expect you to get better
numbers if you increase it to 100 or 250 (probably won't make much
difference above that).
>
> If I have multiple receivers subscribed to a given topic and they each have
> some X amount of credits, how does the sender account for this when sending?
> Some receivers could be slower in replenishing credits than others and the
> sender only sends to the topic and has no idea how many receivers there are.
> After further looking at the examples and AMQP 1.0 spec, it doesn't seem
> like I have much ability at all to control credits from the sender side at
> all. Do you have anything I can look at to learn more about the flow
> control feature?
Credit is always issued by the process receiving messages. So for the
senders, it is the router or the broker that will grant them credit. The
broker and the router behave in quite different ways, but as you are
seeing the same low rate in each case I don't think it is the server
side that is the issue.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: default limits of qpid-c++ broker, dispatch router, or proton?
Posted by tomt <tt...@netapp.com>.
Thanks for the fast response. I spent a good part of the afternoon looking
into the whole flow control system to understand better given what you had
asked.
I am using the Python client as my receivers and the C++ API as my senders
who each synchronously send on their own threads as fast as possible.
I dug a little deeper and found the Python client starts off receivers with
10 credits as the default since I have not taken credits into account for
anything I've done so far. I am having an assumption here that increasing
the number will increase my throughput, but I think I still lack some
general understanding here.
If I have multiple receivers subscribed to a given topic and they each have
some X amount of credits, how does the sender account for this when sending?
Some receivers could be slower in replenishing credits than others and the
sender only sends to the topic and has no idea how many receivers there are.
After further looking at the examples and AMQP 1.0 spec, it doesn't seem
like I have much ability at all to control credits from the sender side at
all. Do you have anything I can look at to learn more about the flow
control feature?
Thanks
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: default limits of qpid-c++ broker, dispatch router, or proton?
Posted by Gordon Sim <gs...@redhat.com>.
On 09/07/2020 3:43 pm, tomt wrote:
> Hello,
>
> I have been trying to determine the highest rate of messages that can be
> sent at any given time through my environment that includes the qpid-cpp
> broker and the dispatch router all using AMQP 1.0 (via Proton).
>
> There seems to be some maximum cap that only lets me send up to 2500
> messages/sec and I'm trying to determine if that's a limitation of my
> environment or perhaps a default in any of the mentioned packages. I have 2
> senders blasting messages and 10 receivers subscribing to all of them.
>
> I have searched around and it looks like any sort of limiting is off by
> default in the broker and router, but that 2500 number is consistent and
> makes me think that there could be a limiter somewhere. Any sort of
> pointers or suggestions would be greatly appreciated.
That is a very low rate (I'm assuming they are not persistent as you get
the same from router as well as broker).
What client are you testing with? Are the receivers only issuing one
credit at a time? Or is the sender publishing synchronously?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: default limits of qpid-c++ broker, dispatch router, or proton?
Posted by tomt <tt...@netapp.com>.
It's been a while, but I did a test with just the proton C++ and QpidJMS
clients and can get closer to Virgilio's numbers.
I think the trouble I was running into was only in the Python Proton
library. For whatever reason, it is far slower than the others.
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: default limits of qpid-c++ broker, dispatch router, or proton?
Posted by Virgilio Fornazin <vi...@gmail.com>.
we've managed to do ~780000 msg /s on qpid-1.36 on 3 qpid broker instances
on a intel X5670, 64GB ram, 4x 1gb lan, 16 clients flooding, 16 clients
receiving (~48000msg /s on each direction).
on a test running on 2011.
I'm pretty sure C++ broker is able to do a bit more than your numbers in
AMQP-0-10
On Thu, Jul 9, 2020 at 11:43 AM tomt <tt...@netapp.com> wrote:
> Hello,
>
> I have been trying to determine the highest rate of messages that can be
> sent at any given time through my environment that includes the qpid-cpp
> broker and the dispatch router all using AMQP 1.0 (via Proton).
>
> There seems to be some maximum cap that only lets me send up to 2500
> messages/sec and I'm trying to determine if that's a limitation of my
> environment or perhaps a default in any of the mentioned packages. I have
> 2
> senders blasting messages and 10 receivers subscribing to all of them.
>
> I have searched around and it looks like any sort of limiting is off by
> default in the broker and router, but that 2500 number is consistent and
> makes me think that there could be a limiter somewhere. Any sort of
> pointers or suggestions would be greatly appreciated.
>
> Thank you.
>
>
>
> --
> Sent from:
> http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>