You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Filipe Santos <Fi...@coollink.pt> on 2014/08/08 16:47:19 UTC
Bulk binding
Hi
I'm trying to bind several queues at the same time in C++.
What I'm presently doing is iterating through a list, and send a biding message per service.
This is quite costly since I have to send the massage and fetch the reply for each one, if I do it sequentially I end up waiting a lot for huge amount of bindings ( I'm trying with 15000).
Is there any way to join all the bindings into a list and send the massive bind request?
I know if there is an error in a bulk I won't know which one has gone wrong, but let's assume that I don't care about that. I also know that I don't need to do them sequentially, I'm using threads and others techniques in order to make it faster, but due to the amount of request it's still slow.
Best regards
Filipe Santos
Re: Bulk binding
Posted by Gordon Sim <gs...@redhat.com>.
On 08/08/2014 06:11 PM, Filipe Santos wrote:
> I got an average of 25 seconds in the request for "create" messages and 22 for "delete" messages (this time does not contains the time of building the message just the send) and 2 seconds in the response, both of them.
That's 25 seconds to send 15,000 create binding messages (without
waiting for responses), right? I.e. about 600 bindings per second. And
that's timed from the just before invoking the first send to the return
of the last send? Did you set the sender capacity at all?
Certainly I don't think the message transfer throughput is the issue
there, so squishing all the requests into a single message is unlikely
to help in my opinion.
I don't think I've ever measured the time for a bind, so I'm not sure
whether 600 binds per second is good or bad. I'll try and do a little
experimentation myself. What type of exchange is it (or is it mixed types)?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
RE: Bulk binding
Posted by Filipe Santos <Fi...@coollink.pt>.
Hi
I got an average of 25 seconds in the request for "create" messages and 22 for "delete" messages (this time does not contains the time of building the message just the send) and 2 seconds in the response, both of them. I found out the best performance was with the send.setCapacity(-1) option enabled. Also tried with {link: {reliability: unreliable}} in the address but haven't got better results.
The broker was our not being used or in low usage, it's a dedicated server that only my team uses. No other tests where being done.
Best,
Filipe Santos
-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com]
Sent: sexta-feira, 8 de Agosto de 2014 17:04
To: users@qpid.apache.org
Subject: Re: Bulk binding
On 08/08/2014 04:57 PM, Filipe Santos wrote:
> I'm doing exactly what you suggested and also threading my requests in order to take advantage of the local cpu, but I'm always eager to improve benchmarks.
How long is it taking doing all the requests asynchronously?
On the fetch() side you have a non-zero capacity on your receiver? Is there anything else going on on the broker while you are binding? E.g.
publishing to the exchange to which the bindings are being established?
Binding hasn't been considered critical path, so its not the most optimised path. If the issue is simply how quickly the broker can establish the bindings, then it may not help to merge them all into one message anyway.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For additional commands, e-mail: users-help@qpid.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Bulk binding
Posted by Gordon Sim <gs...@redhat.com>.
On 08/08/2014 04:57 PM, Filipe Santos wrote:
> I'm doing exactly what you suggested and also threading my requests in order to take advantage of the local cpu, but I'm always eager to improve benchmarks.
How long is it taking doing all the requests asynchronously?
On the fetch() side you have a non-zero capacity on your receiver? Is
there anything else going on on the broker while you are binding? E.g.
publishing to the exchange to which the bindings are being established?
Binding hasn't been considered critical path, so its not the most
optimised path. If the issue is simply how quickly the broker can
establish the bindings, then it may not help to merge them all into one
message anyway.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
RE: Bulk binding
Posted by Filipe Santos <Fi...@coollink.pt>.
Hi
Thank you for your quick answer.
Yes, I was talking about a QMFv2 message to the c++ broker!
I'm doing exactly what you suggested and also threading my requests in order to take advantage of the local cpu, but I'm always eager to improve benchmarks.
Best,
Filipe Santos
-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com]
Sent: sexta-feira, 8 de Agosto de 2014 16:34
To: users@qpid.apache.org
Subject: Re: Bulk binding
On 08/08/2014 03:47 PM, Filipe Santos wrote:
> Hi
>
> I'm trying to bind several queues at the same time in C++.
> What I'm presently doing is iterating through a list, and send a biding message per service.
I'm assuming that is a QMFv2 message to the c++ broker, is that right?
> This is quite costly since I have to send the massage and fetch the reply for each one, if I do it sequentially I end up waiting a lot for huge amount of bindings ( I'm trying with 15000).
> Is there any way to join all the bindings into a list and send the massive bind request?
You know you don't have to fetch the response for each message, right?
You can send all 15,000 requests then process all 15,000 responses. Each response can be correlated to the specific request it refers to.
That would be my first suggestion. I'm not sure there would be an obvious benefit from merging these all into a single request. There is a little bit of overhead, but I wouldn't expect it to be the most significant factor. (I could of course be wrong!)
> I know if there is an error in a bulk I won't know which one has gone wrong, but let's assume that I don't care about that. I also know that I don't need to do them sequentially, I'm using threads and others techniques in order to make it faster, but due to the amount of request it's still slow.
>
>
> Best regards
> Filipe Santos
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For additional commands, e-mail: users-help@qpid.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Bulk binding
Posted by Gordon Sim <gs...@redhat.com>.
On 08/08/2014 03:47 PM, Filipe Santos wrote:
> Hi
>
> I'm trying to bind several queues at the same time in C++.
> What I'm presently doing is iterating through a list, and send a biding message per service.
I'm assuming that is a QMFv2 message to the c++ broker, is that right?
> This is quite costly since I have to send the massage and fetch the reply for each one, if I do it sequentially I end up waiting a lot for huge amount of bindings ( I'm trying with 15000).
> Is there any way to join all the bindings into a list and send the massive bind request?
You know you don't have to fetch the response for each message, right?
You can send all 15,000 requests then process all 15,000 responses. Each
response can be correlated to the specific request it refers to.
That would be my first suggestion. I'm not sure there would be an
obvious benefit from merging these all into a single request. There is a
little bit of overhead, but I wouldn't expect it to be the most
significant factor. (I could of course be wrong!)
> I know if there is an error in a bulk I won't know which one has gone wrong, but let's assume that I don't care about that. I also know that I don't need to do them sequentially, I'm using threads and others techniques in order to make it faster, but due to the amount of request it's still slow.
>
>
> Best regards
> Filipe Santos
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org