You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Kiss PĂŠter <ko...@freemail.hu> on 2010/04/07 11:04:20 UTC
Multithreaded request/reply server in python
Hi everyone,
I'm quite new to Qpid and AMQP, and trying to figure out,
how to write a robust multithreaded request/reply server and
a client in python. (The long term goal is to publish an API
with apache avro RCP using AMQP as a transport.)
I'm using the v0.6 Qpid Java broker and the enclosed python library.
The python API documentation is very poor, the examples are quite heplful,
but too simple. Browsing the code, I found the peer.py and client.py, but couldn't find an example how to use them in a real application.
Any help or code snippet would be much appreciated.
Or, if I'm digging in the wrong direction, please point me to the right way.
I'm aware of the fact, that there are more AMQP libraries for python,
(txamqp, amqplib), but none of them seems to be as up to date as qpid.
(Actually txamqp seems great, but using twisted is a bit frightening.)
Thanks
kodiak
<a href="" target="_blank"><br><br>________________________________________________________<br><a href="http://ad.adverticum.net/b/cl,1,73468,1595374,1595372/click.prm">Nálunk a nyelvtanfolyamok garantáltan elindulnak! Jelentkezz 20% kedvezménnyel a 20 éves Katedra Budapesthez április 7-9. között!</a></a>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Multithreaded request/reply server in python
Posted by Cajus Pollmeier <ca...@naasa.net>.
Hiho,
better late than never - I finally found time to take a look at it: thanks
for the code!
I've played a little bit with the threading and tried to measure the
performance of the Python client/server - and it seems that I'm doing
something completely wrong ;-)
If I use your previous mail example as the req/response server and this
micro snippet
8<-----------
from qpid.messaging import *
conn = Connection.establish("localhost")
ssn = conn.session(str(uuid4()))
snd = ssn.sender("requests")
rcv = ssn.receiver('reply-%s; {create:always, delete:always}' % ssn.name)
for i in range(1000):
msg = Message("The quick brown fox tests the performance")
msg.reply_to = 'reply-%s' % ssn.name
snd.send(msg)
res = rcv.fetch()
print res.content
conn.close()
8<-----------
, it takes ~30 seconds to process 1000 messages on the local machine. The
server eats 25% of my CPU and qpid itself seems to be bored (0%).
Using the profiler, I can see that python seems to wait for locks for over
90% of its processing time. Is this a GIL issue, or do I need some tweaking
in the way threads get used?
Cheers,
Cajus
On Tue, 13 Apr 2010 15:09:44 -0400, Rafael Schloming <ra...@redhat.com>
wrote:
> I've attached one way to do a multi-threaded request/reply server. There
> are others as well. Is this the sort of thing you're looking for?
>
> Note that this uses the latest version of the API from trunk which does
> have a few naming changes relative to the 0.6 version, however this
> should work on the 0.6 version as will with some minor tweaks.
>
> --Rafael
>
> Cajus Pollmeier wrote:
>> I'm facing a similar problem. Maybe it would be a good thing to move an
>> example to the wiki to allow others a more rapid coding start.
>>
>> My 5 cent,
>> Cajus
>>
>> Am 08.04.2010 um 15:26 schrieb Carl Trieloff:
>>>
>>> This can be done with the qpid python libs, there are a few threads on
>>> the list already on the topic.
>>>
>>> If you can't locate them in the archive, shout, and we can maybe post
>>> an example and post it on the
>>> wiki.
>>>
>>> Carl.
>>>
>>>
>>> On 04/07/2010 05:04 AM, Kiss Péter wrote:
>>>> Hi everyone,
>>>>
>>>> I'm quite new to Qpid and AMQP, and trying to figure out,
>>>> how to write a robust multithreaded request/reply server and
>>>> a client in python. (The long term goal is to publish an API
>>>> with apache avro RCP using AMQP as a transport.)
>>>> I'm using the v0.6 Qpid Java broker and the enclosed python library.
>>>> The python API documentation is very poor, the examples are quite
>>>> heplful,
>>>> but too simple. Browsing the code, I found the peer.py and client.py,
>>>> but couldn't find an example how to use them in a real application.
>>>> Any help or code snippet would be much appreciated.
>>>> Or, if I'm digging in the wrong direction, please point me to the
>>>> right way.
>>>>
>>>> I'm aware of the fact, that there are more AMQP libraries for python,
>>>> (txamqp, amqplib), but none of them seems to be as up to date as
qpid.
>>>> (Actually txamqp seems great, but using twisted is a bit
frightening.)
>>>>
>>>> Thanks
>>>> kodiak
>>>>
>>>> <a href=""
>>>>
target="_blank"><br><br>________________________________________________________<br><a
>>>>
href="http://ad.adverticum.net/b/cl,1,73468,1595374,1595372/click.prm">Nálunk
>>>> a nyelvtanfolyamok garantáltan elindulnak! Jelentkezz 20%
>>>> kedvezménnyel a 20 éves Katedra Budapesthez április 7-9.
>>>> között!</a></a>
>>>>
>>>> ---------------------------------------------------------------------
>>>> Apache Qpid - AMQP Messaging Implementation
>>>> Project: http://qpid.apache.org
>>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> Apache Qpid - AMQP Messaging Implementation
>>> Project: http://qpid.apache.org
>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Multithreaded request/reply server in python
Posted by Rafael Schloming <ra...@redhat.com>.
I've attached one way to do a multi-threaded request/reply server. There
are others as well. Is this the sort of thing you're looking for?
Note that this uses the latest version of the API from trunk which does
have a few naming changes relative to the 0.6 version, however this
should work on the 0.6 version as will with some minor tweaks.
--Rafael
Cajus Pollmeier wrote:
> I'm facing a similar problem. Maybe it would be a good thing to move an example to the wiki to allow others a more rapid coding start.
>
> My 5 cent,
> Cajus
>
> Am 08.04.2010 um 15:26 schrieb Carl Trieloff:
>>
>> This can be done with the qpid python libs, there are a few threads on the list already on the topic.
>>
>> If you can't locate them in the archive, shout, and we can maybe post an example and post it on the
>> wiki.
>>
>> Carl.
>>
>>
>> On 04/07/2010 05:04 AM, Kiss Péter wrote:
>>> Hi everyone,
>>>
>>> I'm quite new to Qpid and AMQP, and trying to figure out,
>>> how to write a robust multithreaded request/reply server and
>>> a client in python. (The long term goal is to publish an API
>>> with apache avro RCP using AMQP as a transport.)
>>> I'm using the v0.6 Qpid Java broker and the enclosed python library.
>>> The python API documentation is very poor, the examples are quite heplful,
>>> but too simple. Browsing the code, I found the peer.py and client.py, but couldn't find an example how to use them in a real application.
>>> Any help or code snippet would be much appreciated.
>>> Or, if I'm digging in the wrong direction, please point me to the right way.
>>>
>>> I'm aware of the fact, that there are more AMQP libraries for python,
>>> (txamqp, amqplib), but none of them seems to be as up to date as qpid.
>>> (Actually txamqp seems great, but using twisted is a bit frightening.)
>>>
>>> Thanks
>>> kodiak
>>>
>>> <a href="" target="_blank"><br><br>________________________________________________________<br><a href="http://ad.adverticum.net/b/cl,1,73468,1595374,1595372/click.prm">Nálunk a nyelvtanfolyamok garantáltan elindulnak! Jelentkezz 20% kedvezménnyel a 20 éves Katedra Budapesthez április 7-9. között!</a></a>
>>>
>>> ---------------------------------------------------------------------
>>> Apache Qpid - AMQP Messaging Implementation
>>> Project: http://qpid.apache.org
>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
Re: Multithreaded request/reply server in python
Posted by Cajus Pollmeier <ca...@naasa.net>.
I'm facing a similar problem. Maybe it would be a good thing to move an example to the wiki to allow others a more rapid coding start.
My 5 cent,
Cajus
Am 08.04.2010 um 15:26 schrieb Carl Trieloff:
>
>
> This can be done with the qpid python libs, there are a few threads on the list already on the topic.
>
> If you can't locate them in the archive, shout, and we can maybe post an example and post it on the
> wiki.
>
> Carl.
>
>
> On 04/07/2010 05:04 AM, Kiss Péter wrote:
>> Hi everyone,
>>
>> I'm quite new to Qpid and AMQP, and trying to figure out,
>> how to write a robust multithreaded request/reply server and
>> a client in python. (The long term goal is to publish an API
>> with apache avro RCP using AMQP as a transport.)
>> I'm using the v0.6 Qpid Java broker and the enclosed python library.
>> The python API documentation is very poor, the examples are quite heplful,
>> but too simple. Browsing the code, I found the peer.py and client.py, but couldn't find an example how to use them in a real application.
>> Any help or code snippet would be much appreciated.
>> Or, if I'm digging in the wrong direction, please point me to the right way.
>>
>> I'm aware of the fact, that there are more AMQP libraries for python,
>> (txamqp, amqplib), but none of them seems to be as up to date as qpid.
>> (Actually txamqp seems great, but using twisted is a bit frightening.)
>>
>> Thanks
>> kodiak
>>
>> <a href="" target="_blank"><br><br>________________________________________________________<br><a href="http://ad.adverticum.net/b/cl,1,73468,1595374,1595372/click.prm">Nálunk a nyelvtanfolyamok garantáltan elindulnak! Jelentkezz 20% kedvezménnyel a 20 éves Katedra Budapesthez április 7-9. között!</a></a>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Multithreaded request/reply server in python
Posted by Carl Trieloff <cc...@redhat.com>.
This can be done with the qpid python libs, there are a few threads on
the list already on the topic.
If you can't locate them in the archive, shout, and we can maybe post an
example and post it on the
wiki.
Carl.
On 04/07/2010 05:04 AM, Kiss Péter wrote:
> Hi everyone,
>
> I'm quite new to Qpid and AMQP, and trying to figure out,
> how to write a robust multithreaded request/reply server and
> a client in python. (The long term goal is to publish an API
> with apache avro RCP using AMQP as a transport.)
> I'm using the v0.6 Qpid Java broker and the enclosed python library.
> The python API documentation is very poor, the examples are quite heplful,
> but too simple. Browsing the code, I found the peer.py and client.py, but couldn't find an example how to use them in a real application.
> Any help or code snippet would be much appreciated.
> Or, if I'm digging in the wrong direction, please point me to the right way.
>
> I'm aware of the fact, that there are more AMQP libraries for python,
> (txamqp, amqplib), but none of them seems to be as up to date as qpid.
> (Actually txamqp seems great, but using twisted is a bit frightening.)
>
> Thanks
> kodiak
>
> <a href="" target="_blank"><br><br>________________________________________________________<br><a href="http://ad.adverticum.net/b/cl,1,73468,1595374,1595372/click.prm">Nálunk a nyelvtanfolyamok garantáltan elindulnak! Jelentkezz 20% kedvezménnyel a 20 éves Katedra Budapesthez április 7-9. között!</a></a>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org