Re: Multithreaded request/reply server in python

[Cajus Pollmeier <ca...@naasa.net>]

Hiho,

better late than never - I finally found time to take a look at it: thanks
for the code!

I've played a little bit with the threading and tried to measure the
performance of the Python client/server - and it seems that I'm doing
something completely wrong ;-)

If I use your previous mail example as the req/response server and this
micro snippet

8<-----------
from qpid.messaging import *

conn = Connection.establish("localhost")
ssn = conn.session(str(uuid4()))
snd = ssn.sender("requests")
rcv = ssn.receiver('reply-%s; {create:always, delete:always}' % ssn.name)

for i in range(1000):
    msg = Message("The quick brown fox tests the performance")
    msg.reply_to = 'reply-%s' % ssn.name
    snd.send(msg)
    res = rcv.fetch()
    print res.content

conn.close()
8<-----------

, it takes ~30 seconds to process 1000 messages on the local machine. The
server eats 25% of my CPU and qpid itself seems to be bored (0%).

Using the profiler, I can see that python seems to wait for locks for over
90% of its processing time. Is this a GIL issue, or do I need some tweaking
in the way threads get used?

Cheers,
Cajus


On Tue, 13 Apr 2010 15:09:44 -0400, Rafael Schloming <ra...@redhat.com>
wrote:
> I've attached one way to do a multi-threaded request/reply server. There

>   are others as well. Is this the sort of thing you're looking for?
> 
> Note that this uses the latest version of the API from trunk which does 
> have a few naming changes relative to the 0.6 version, however this 
> should work on the 0.6 version as will with some minor tweaks.
> 
> --Rafael
> 
> Cajus Pollmeier wrote:
>> I'm facing a similar problem. Maybe it would be a good thing to move an
>> example to the wiki to allow others a more rapid coding start.
>> 
>> My 5 cent,
>> Cajus
>> 
>> Am 08.04.2010 um 15:26 schrieb Carl Trieloff:
>>>
>>> This can be done with the qpid python libs, there are a few threads on
>>> the list already on the topic.
>>>
>>> If you can't locate them in the archive, shout, and we can maybe post
>>> an example and post it on the
>>> wiki.
>>>
>>> Carl.
>>>
>>>
>>> On 04/07/2010 05:04 AM, Kiss Péter wrote:
>>>> Hi everyone,
>>>>
>>>> I'm quite new to Qpid and AMQP, and trying to figure out,
>>>> how to write a robust multithreaded request/reply server and
>>>> a client in python. (The long term goal is to publish an API
>>>> with apache avro RCP using AMQP as a transport.)
>>>> I'm using the v0.6 Qpid Java broker and the enclosed python library.
>>>> The python API documentation is very poor, the examples are quite
>>>> heplful,
>>>> but too simple. Browsing the code, I found the peer.py and client.py,
>>>> but couldn't find an example how to use them in a real application.
>>>> Any help or code snippet would be much appreciated.
>>>> Or, if I'm digging in the wrong direction, please point me to the
>>>> right way.
>>>>
>>>> I'm aware of the fact, that there are more AMQP libraries for python,
>>>> (txamqp, amqplib), but none of them seems to be as up to date as
qpid.
>>>> (Actually txamqp seems great, but using twisted is a bit
frightening.)
>>>>
>>>> Thanks
>>>> kodiak
>>>>
>>>> <a href=""
>>>>
target="_blank"><br><br>________________________________________________________<br><a
>>>>
href="http://ad.adverticum.net/b/cl,1,73468,1595374,1595372/click.prm">Nálunk
>>>> a nyelvtanfolyamok garantáltan elindulnak! Jelentkezz 20%
>>>> kedvezménnyel a 20 éves Katedra Budapesthez április 7-9.
>>>> között!</a></a>
>>>>
>>>> ---------------------------------------------------------------------
>>>> Apache Qpid - AMQP Messaging Implementation
>>>> Project:      http://qpid.apache.org
>>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>>
>>>>   
>>>
>>> ---------------------------------------------------------------------
>>> Apache Qpid - AMQP Messaging Implementation
>>> Project:      http://qpid.apache.org
>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>
>> 
>> 
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project:      http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org