txAMQP released

[Esteve Fernandez <es...@sindominio.net>]

Hi all,

I'm pleased to announce that Fluidinfo [1] is releasing a Twisted [2]
compatible AMQP stack built on the Python Qpid library.

Download from http://foss.fluidinfo.com

We've been using it for a while at Fluidinfo and have converted all the
Qpid tests to use it, so it should be considered fairly stable.

Although we'll try to keep it in sync with Qpid's stable releases, we can't
promise to track the trunk due to lack of resources.

Feedback and contributions will be greatly appreciated! :-)

Cheers.

PS: I don't know which license is the most appropiate to make it compatible
with Qpid, so I guess the ASL should suffice.

[1] - Fluidinfo - http://www.fluidinfo.com
[2] - Twisted - http://www.twistedmatrix.com

Re: metadata describing message properties?

[Robert Greig <ro...@gmail.com>]

2008/7/22 Shahbaz Chaudhary <sc...@genghisfund.com>:
> There is a getPropertyNames() function, but there doesn't seem to be one
> which returns the types of the fields.
>
> Secondly, the property names are delivered as part of a message, does
> this information exist on a destination level.  In other words, before a
> message is received, does (or can) the client known such meta data?

No, the properties are entirely in the application domain. You would
have to build something that returned a message containing the details
of the properties.

> My motivation is a generic Jtable which can be linked to a destination
> (topic or queue).  The JTable model needs to know the number of fields
> and their types, before a message is received.

The number of fields and their types is something your application
specifies. I'd create a Service queue as follows:

1) server registers a service listening on the Service queue
2) client creates a private queue
3) client sends a message to the Service queue with a property
indicating the operation is "queryProperties", and sets the ReplyTo
address to the private queue
4) server sends a message (in XML say) that contains the topics plus
the property names and types to the client's private queue
5) client can parse the message and do with it what it wants

You can obviously build in as much sophistication as you like into the services.

RG

RE: metadata describing message properties?

[Shahbaz Chaudhary <sc...@genghisfund.com>]

There is a getPropertyNames() function, but there doesn't seem to be one
which returns the types of the fields.

Secondly, the property names are delivered as part of a message, does
this information exist on a destination level.  In other words, before a
message is received, does (or can) the client known such meta data?

My motivation is a generic Jtable which can be linked to a destination
(topic or queue).  The JTable model needs to know the number of fields
and their types, before a message is received.


-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com] 
Sent: Tuesday, July 22, 2008 4:14 AM
To: qpid-users@incubator.apache.org
Subject: Re: metadata describing message properties?

Shahbaz Chaudhary wrote:
> I didn't notice the ability to retreive the names and types of
> properties contained in messages in the JMS api.  Is this something
> qpid supports already or plans on supporting?  This is very similar
> to the metadata provided by the JDBC api.

I'm not entirely sure I'm answering the right question here, but JMS 
does indeed offer the ability to get and set various message properties 
and the Qpid JMS client implements them. See the javadoc for the Message

interface [1].

[1] http://java.sun.com/j2ee/1.4/docs/api/javax/jms/Message.html

Re: metadata describing message properties?

[Gordon Sim <gs...@redhat.com>]

Shahbaz Chaudhary wrote:
> I didn't notice the ability to retreive the names and types of
> properties contained in messages in the JMS api.  Is this something
> qpid supports already or plans on supporting?  This is very similar
> to the metadata provided by the JDBC api.

I'm not entirely sure I'm answering the right question here, but JMS 
does indeed offer the ability to get and set various message properties 
and the Qpid JMS client implements them. See the javadoc for the Message 
interface [1].

[1] http://java.sun.com/j2ee/1.4/docs/api/javax/jms/Message.html

metadata describing message properties?

[Shahbaz Chaudhary <sc...@genghisfund.com>]

I didn't notice the ability to retreive the names and types of properties contained in messages in the JMS api.  Is this something qpid supports already or plans on supporting?  This is very similar to the metadata provided by the JDBC api.

The idea is that I want to type out text name of a destination, and programatically retreive the type of the message, bind it to a Jtable, etc.

Perhaps later, I would look to connect to a topic, and retrieve the complete list of destinations I havve access to, and continue on with allowing users to click on an automatically pre-populated list of destination and autogenerate jtables to view the data.

Thanks

Re: java.lang.OutOfMemoryError: Java heap space

[Robert Greig <ro...@gmail.com>]

2008/7/24 Shahbaz Chaudhary <sc...@genghisfund.com>:

> As mentioned in a previous email, the only reason for this I can think of is that the queues are not getting dequeued fast enough (the clients are too slow) and they start to fill up.
>
> I spent some time going through the JMX monitor (VisualJVM), but, frankly, I can't tell which queues are the ones taking up all the memory.

It would be useful to know if you have any queues with messages on
them. It doesn't really matter at this stage whether you can tell
which queue is the problem - having queues with lots of messages on
them will at least help us rule out other types of bug.

> How can I resolve this?

How many clients do you have? I would start with one client and check
the queue depth(s), and increase the number of clients.

> If C# clients were keeping up with the trunk version of qpid, I'd just upgrade and see if the issue gets resolved :)

There is an M2.1 release which is newer, however right now I would try
to do some diagnosis before upgrading anything.

Robert

java.lang.OutOfMemoryError: Java heap space

[Shahbaz Chaudhary <sc...@genghisfund.com>]

I have recently been dealing with qpid (the last release M2 version) running out of memory.  I'm running it on a linux box with 4 gigs of ram (2 gigs allocated to qpid).

I start sending data to qpid around 9:30 through the C# api and qpid starts with about 10 megs of memory use.  Today, around 3:30, it used up its two gigs.

As mentioned in a previous email, the only reason for this I can think of is that the queues are not getting dequeued fast enough (the clients are too slow) and they start to fill up.

I spent some time going through the JMX monitor (VisualJVM), but, frankly, I can't tell which queues are the ones taking up all the memory.

I have an hprof file (1.5 gig) as well which is generated when the jvm dies due to memory problems.  I can't quite tell what to look for in the file either.

Finally, I don’t think my clients are too slow.  The largest amount of data I publish is less than 150 megs!

How can I resolve this?  

If C# clients were keeping up with the trunk version of qpid, I'd just upgrade and see if the issue gets resolved :)

Re: txAMQP released

[Esteve Fernandez <es...@sindominio.net>]

El Miércoles 23 Julio 2008 02:14:52 Carl Trieloff escribió:
> what page do you want me to link?

http://launchpad.net/txamp is the project page. BTW, would it be possible to 
add a link to http://fluidinfo.com as well to get some publicity?

Cheers.

Re: txAMQP released

[Carl Trieloff <cc...@redhat.com>]

Esteve Fernandez wrote:
> Hi Carl
>
> El Martes 22 Julio 2008 13:37:56 Carl Trieloff escribió:
>   
>> if you create a blurb about twisted, we can add it as an integration to
>> the wiki.
>>     
>
> What about this (taken from the Twisted website):
>
> "Twisted is an event-driven network programming framework written in Python 
> and licensed under the MIT License."
>
> and here's a blurb for txAMQP:
>
> "txAMQP is a Python library for communicating with AMQP peers and brokers 
> using Twisted"
>
> Cheers.
>   

what page do you want me to link?

Carl.

Re: txAMQP released

[Esteve Fernandez <es...@sindominio.net>]

Hi Carl

El Martes 22 Julio 2008 13:37:56 Carl Trieloff escribió:
> if you create a blurb about twisted, we can add it as an integration to
> the wiki.

What about this (taken from the Twisted website):

"Twisted is an event-driven network programming framework written in Python 
and licensed under the MIT License."

and here's a blurb for txAMQP:

"txAMQP is a Python library for communicating with AMQP peers and brokers 
using Twisted"

Cheers.

Re: txAMQP released

[Carl Trieloff <cc...@redhat.com>]

if you create a blurb about twisted, we can add it as an integration to 
the wiki.

Carl.


Esteve Fernandez wrote:
> El Lunes 21 Julio 2008 21:40:19 Robert Greig escribió:
>   
>> Hi,
>>     
>
> Hi Robert
>
>   
>> It's great that you are building on top of qpid.
>>     
>
> Thanks, we found the Qpid Python stack very well designed, we were able to 
> plug our Twisted implementation without too much effort.
>
>   
>> For those of us who live in the stone age and don't know much about
>> Twisted can you give us a quick summary of what an AMQP user gets from
>> using Twisted?
>>     
>
> Well, Twisted departs from the traditional synchronous way of building network 
> applications, through an asynchronous API, making it very scalable. One of 
> the main advantages is that you don't need to deal with threads and their 
> implying risks. Since the GIL imposes both memory and performance penalties 
> if you use many threads (i.e. for reading from a socket), using an 
> asynchronous framework (such as Twisted) will help you build very scalable 
> architectures.
>
> However, having to code in an asynchronous way is not straightforward, and if 
> you already have some code that uses recv/send (or some other synchronous 
> blocking functions), you might find it a bit hard to convert it to use 
> Twisted. There are some facilities to ease the transition, such as the 
> inlineCallbacks decorator, which we used heavily to convert the Qpid tests to 
> use txAMQP.
>
>   
>> If I am a Python developer and want to use Qpid or AMQP, should I be
>> downloading Twisted?
>>     
>
> As always, it depends on the problem you want to solve. If it's a simple 
> client, that it's going to execute once in a while, maybe not. But, if your 
> problem is complex and needs to be scalable, I recommend you to give Twisted 
> a try.
>
> Cheers.
>   

Re: txAMQP released

[Esteve Fernandez <es...@sindominio.net>]

El Lunes 21 Julio 2008 21:40:19 Robert Greig escribió:
> Hi,

Hi Robert

> It's great that you are building on top of qpid.

Thanks, we found the Qpid Python stack very well designed, we were able to 
plug our Twisted implementation without too much effort.

> For those of us who live in the stone age and don't know much about
> Twisted can you give us a quick summary of what an AMQP user gets from
> using Twisted?

Well, Twisted departs from the traditional synchronous way of building network 
applications, through an asynchronous API, making it very scalable. One of 
the main advantages is that you don't need to deal with threads and their 
implying risks. Since the GIL imposes both memory and performance penalties 
if you use many threads (i.e. for reading from a socket), using an 
asynchronous framework (such as Twisted) will help you build very scalable 
architectures.

However, having to code in an asynchronous way is not straightforward, and if 
you already have some code that uses recv/send (or some other synchronous 
blocking functions), you might find it a bit hard to convert it to use 
Twisted. There are some facilities to ease the transition, such as the 
inlineCallbacks decorator, which we used heavily to convert the Qpid tests to 
use txAMQP.

> If I am a Python developer and want to use Qpid or AMQP, should I be
> downloading Twisted?

As always, it depends on the problem you want to solve. If it's a simple 
client, that it's going to execute once in a while, maybe not. But, if your 
problem is complex and needs to be scalable, I recommend you to give Twisted 
a try.

Cheers.

Re: txAMQP released

[Robert Greig <ro...@gmail.com>]

2008/7/21 Esteve Fernandez <es...@sindominio.net>:

> I'm pleased to announce that Fluidinfo [1] is releasing a Twisted [2]
> compatible AMQP stack built on the Python Qpid library.

Hi,

It's great that you are building on top of qpid.

For those of us who live in the stone age and don't know much about
Twisted can you give us a quick summary of what an AMQP user gets from
using Twisted?

If I am a Python developer and want to use Qpid or AMQP, should I be
downloading Twisted?

Thanks,

Robert