You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Anthony Foglia <AF...@princeton.com> on 2011/06/03 20:02:40 UTC
Recommended API for Python?
What are the differences between the three Python APIs? There are (at
least) 3 APIs:
1. qpid package
This was the first API we used, based on some examples in a two year
old blog post.
2. qpid.messaging package
This one more closely resembles the C++ API, and from playing around,
appears to have more knobs, like setting the receiver capacity.
3. SWIG api
This one is a wrapping of the C++ library, but unfortunately is
nowhere near a drop in replacement for the qpid.messaging API. Also, in
my simple timings (of a consumer with receiver capacity of 1000,
asynchronous acknowledging receipts in batches of 10000), I didn't see
any performance gains compared to the qpid.messaging library.
What is the consensus suggestion for a new project requiring
high-throughput?
--
Anthony Foglia
Princeton Consultants
(609) 987-8787 x233
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Recommended API for Python?
Posted by Gordon Sim <gs...@redhat.com>.
On 06/09/2011 03:48 PM, AFoglia@princeton.com wrote:
> On 6/9/2011 6:37 AM, Gordon Sim wrote:
>> I would also echo Ted's comments, that the messaging API is really
>> what we recommend, and that the swigged version when completed
>> would be a drop in replacement for that (avoiding annoyances like
>> the above).
>
> I agree that it should be a drop-in replacement, and I'm willing to
> help make the necessary changes, but I don't see where. In the 0.10
> release, there's the python package,
> qpidc-0.10/bindings/qpid/python/cqpid.py, that would need to be
> patched, but I don't see that in the repository.
It's generated by swig.
> Where would the changes need to go?
I believe that the swig generation process can be customised to a
degree. However I suspect that would not be sufficient to be a drop in
replacement and that therefore some adapter layer would be required.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Recommended API for Python?
Posted by AF...@princeton.com.
On 6/9/2011 6:37 AM, Gordon Sim wrote:
> On 06/08/2011 04:11 PM, Anthony Foglia wrote:
>> On 06/08/2011 09:00 AM, Carl Trieloff wrote:
>>> async transfer is the largest impact, not acknowledgement.
>>
>> How do I do that? I'm already setting the capacity of the receiver to
>> 1000, which we found was the best for the Python code.
>
> I believe the issue is in the way you are doing this for the swigged
> client. At present you need to use the setCapacity() method to set the
> capacity rather than setting it as an attribute.
>
> With that change in place I see the rcv_rate for the swig api jump from
> ~1,300 to ~19,000 for 20,000 msgs.
Thanks. I'm seeing the same changes. I need to change my producer to use the SWIG interface, but now that I can see the speed up in the consumer, I'd expect similar performance in the producer.
> I would also echo Ted's comments, that the messaging API is really what
> we recommend, and that the swigged version when completed would be a
> drop in replacement for that (avoiding annoyances like the above).
I agree that it should be a drop-in replacement, and I'm willing to help make the necessary changes, but I don't see where. In the 0.10 release, there's the python package, qpidc-0.10/bindings/qpid/python/cqpid.py, that would need to be patched, but I don't see that in the repository. Where would the changes need to go?
> [Just for clarity: synchronous acknowledgement per message would have a
> pretty similar effect to synchronous transfer (round trip forced
> per-message). Obviously if the acknowledgements are batched then the
> roundtrip is less frequent and therefore has less impact].
--
Anthony Foglia
Princeton Consultants
(609) 987-8787 x233
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Recommended API for Python?
Posted by Gordon Sim <gs...@redhat.com>.
On 06/08/2011 04:11 PM, Anthony Foglia wrote:
> On 06/08/2011 09:00 AM, Carl Trieloff wrote:
>> async transfer is the largest impact, not acknowledgement.
>
> How do I do that? I'm already setting the capacity of the receiver to
> 1000, which we found was the best for the Python code.
I believe the issue is in the way you are doing this for the swigged
client. At present you need to use the setCapacity() method to set the
capacity rather than setting it as an attribute.
With that change in place I see the rcv_rate for the swig api jump from
~1,300 to ~19,000 for 20,000 msgs.
I would also echo Ted's comments, that the messaging API is really what
we recommend, and that the swigged version when completed would be a
drop in replacement for that (avoiding annoyances like the above).
[Just for clarity: synchronous acknowledgement per message would have a
pretty similar effect to synchronous transfer (round trip forced
per-message). Obviously if the acknowledgements are batched then the
roundtrip is less frequent and therefore has less impact].
>> On 06/07/2011 08:02 PM, Anthony Foglia wrote:
>>> I've attached the code I'm using in a tarball, in case people have any
>>> suggestions.
>>
>>
>> tar's don't attach to the list, use a JIRA.
FWIW the tars came through on the original mail fine for me....
>
> Okay. I opened a JIRA task and attached my code (plus two of the outputs).
>
> https://issues.apache.org/jira/browse/QPID-3297
>
Re: Recommended API for Python?
Posted by Anthony Foglia <AF...@princeton.com>.
On 06/08/2011 09:00 AM, Carl Trieloff wrote:
> async transfer is the largest impact, not acknowledgement.
How do I do that? I'm already setting the capacity of the receiver to
1000, which we found was the best for the Python code. Is there another
knob to ensure that is asynchronous?
I found this page from the Red Hat MRG manual, but (a) it's talking
about a producer, not a consumer, and (b) the casting of a normal
session to an asynchronous session isn't readily available through SWIG.
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.1/html/Messaging_Tutorial/sect-Messaging_Tutorial-Using_RHM_with_C-Optimizing_message_transfer_with_asynchronous_sessions_in_C.html
I haven't even tried speeding up the producer yet; that's still written
using the old Python API.
> On 06/07/2011 08:02 PM, Anthony Foglia wrote:
>> I've attached the code I'm using in a tarball, in case people have any
>> suggestions.
>
>
> tar's don't attach to the list, use a JIRA.
Okay. I opened a JIRA task and attached my code (plus two of the outputs).
https://issues.apache.org/jira/browse/QPID-3297
--
Anthony Foglia
Princeton Consultants
(609) 987-8787 x233
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Recommended API for Python?
Posted by Carl Trieloff <cc...@redhat.com>.
On 06/07/2011 08:02 PM, Anthony Foglia wrote:
> On 06/03/2011 02:58 PM, Ted Ross wrote:
>> API (3) is a literal/mechanical binding of the C++ API in Python. Our
>> intent is to provide a thin Python wrapper around this binding that is a
>> drop-in replacement for API(2). The benefits of this API are that all of
>> the features of the C++ API are made available via Python (RDMA
>> transport, full SASL auth/security support, future features, etc.). I
>> would also expect the wrapped API to perform better than the pure-Python
>> API. I find it surprising that you're not seeing this.
>
> Well, I've repeated my timings numerous times, and the results seem to
> be that the on average the SWIG library performs slightly faster that
> the Python library. But the variance is humongous. I'm only timing
> the consumer side, so maybe the performance will be better on the
> sender. But for my last 5 runs of 20000 messages, I'm getting rates of
> 1600 +/- 20 for the pure python api, and 1320 +/- 844 for the swig
> version. And these are the typical results, comparable times on
> average, with huge variance for the swig api.
>
> One major difference though is that the python version is spending
> nearly all the time in user space, while the swig version is only
> spending a tenth in user, and 5% in system space. I assume the rest
> is waiting for the network, but (a) I'm trying to acknowledge the
> messages asynchronously, and (b) it would be quite the coincidence if
> the Python client is getting maxed at exactly the same rate as the C++
> broker.
async transfer is the largest impact, not acknowledgement.
>
> Either of these times are way less than the 50,000 transfers/sec
> qpid-perftest tells me the broker is capable of. (34,000 publish,
> 26,000 subscribe).
>
> I've attached the code I'm using in a tarball, in case people have any
> suggestions.
tar's don't attach to the list, use a JIRA.
Carl.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Recommended API for Python?
Posted by Anthony Foglia <AF...@princeton.com>.
On 06/03/2011 02:58 PM, Ted Ross wrote:
> API (3) is a literal/mechanical binding of the C++ API in Python. Our
> intent is to provide a thin Python wrapper around this binding that is a
> drop-in replacement for API(2). The benefits of this API are that all of
> the features of the C++ API are made available via Python (RDMA
> transport, full SASL auth/security support, future features, etc.). I
> would also expect the wrapped API to perform better than the pure-Python
> API. I find it surprising that you're not seeing this.
Well, I've repeated my timings numerous times, and the results seem to
be that the on average the SWIG library performs slightly faster that
the Python library. But the variance is humongous. I'm only timing the
consumer side, so maybe the performance will be better on the sender.
But for my last 5 runs of 20000 messages, I'm getting rates of 1600 +/-
20 for the pure python api, and 1320 +/- 844 for the swig version. And
these are the typical results, comparable times on average, with huge
variance for the swig api.
One major difference though is that the python version is spending
nearly all the time in user space, while the swig version is only
spending a tenth in user, and 5% in system space. I assume the rest is
waiting for the network, but (a) I'm trying to acknowledge the messages
asynchronously, and (b) it would be quite the coincidence if the Python
client is getting maxed at exactly the same rate as the C++ broker.
Either of these times are way less than the 50,000 transfers/sec
qpid-perftest tells me the broker is capable of. (34,000 publish,
26,000 subscribe).
I've attached the code I'm using in a tarball, in case people have any
suggestions.
--
Anthony Foglia
Princeton Consultants
(609) 987-8787 x233
Re: Recommended API for Python?
Posted by Ted Ross <tr...@redhat.com>.
On 06/03/2011 02:02 PM, Anthony Foglia wrote:
> What are the differences between the three Python APIs? There are (at
> least) 3 APIs:
>
> 1. qpid package
> This was the first API we used, based on some examples in a two year
> old blog post.
> 2. qpid.messaging package
> This one more closely resembles the C++ API, and from playing
> around, appears to have more knobs, like setting the receiver capacity.
> 3. SWIG api
> This one is a wrapping of the C++ library, but unfortunately is
> nowhere near a drop in replacement for the qpid.messaging API. Also,
> in my simple timings (of a consumer with receiver capacity of 1000,
> asynchronous acknowledging receipts in batches of 10000), I didn't see
> any performance gains compared to the qpid.messaging library.
>
> What is the consensus suggestion for a new project requiring
> high-throughput?
>
API (1) was the original interface for AMQP version 0-10. This API has
a fairly tight coupling to the underlying AMQP protocol and commands.
API (2) was introduced to provide a more abstract interface that will
allow applications to transition to the upcoming 1.0 version of the AMQP
protocol. This is the API that we recommend that you use.
API (3) is a literal/mechanical binding of the C++ API in Python. Our
intent is to provide a thin Python wrapper around this binding that is a
drop-in replacement for API(2). The benefits of this API are that all
of the features of the C++ API are made available via Python (RDMA
transport, full SASL auth/security support, future features, etc.). I
would also expect the wrapped API to perform better than the pure-Python
API. I find it surprising that you're not seeing this.
Regards,
-Ted
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org