You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Fraser Adams <fr...@blueyonder.co.uk> on 2011/12/11 16:52:08 UTC
New Feature Announce - Java QMF2 API Implementation.
Hello All,
By way of giving something back to the Qpid community I've (finally) got
my implementation of the QMF2 API (which for those not already familiar
is described here https://cwiki.apache.org/qpid/qmfv2-api-proposal.html)
into a form that is hopefully generally usable.
It's taken a bit longer than I'd have liked, but I've spent the time
adding some example tools that are based on the API. Hopefully these
tools are useful, at the very least they provide a fair illustration of
the QMF2 API including such things as method invocation and Query
Subscription.
There's quite a lot of if (just short of 100 classes including tests and
tools) so I've zipped it up and attached it to a Jira.
https://issues.apache.org/jira/browse/QPID-3675
Note that there are some quirks, especially if you use a version of Qpid
< 0.12. Older versions require a patch to allow the app-id property name
to be set. The patch is included in the distro and the shell scripts
that launch the tests and tools add the patch jar to the front of the
class path. I've tested with Qpid 0.8, 0.10 and 0.12.
Note too that some of the features won't work with Qpid 0.8. In
particular the QMF2 create and delete methods weren't added until Qpid
0.10 so the add queue/add binding etc. features of QpidConfig won't work
with Qpid 0.8, similarly for QpidCtrl.
The code contains a fair amount of comment and JavaDoc so hopefully is
should be fairly easy to use.
I certainly can't guarantee it'll be bug free :-D but I've tried
reasonably hard to break things, so at the very least it should be
robust enough to be usable. ConnectionAudit and QpidQueueStats seem to
reconnect and survive broker restarts pretty well.
This is the first release of a QMF2 API Implementation for Java.
Features:
* Full Implementation of QMF2 Console, Agent and AgentExternal.
* Supports QMF2 Query Subscriptions on Agent/AgentExternal
implementations.
* Emulates QMF2 Query Subscriptions on the Console side for the
broker ManagementAgent by
intercepting _data indications and filtering against QmfQuery.
This is necessary as the
ManagemetAgent doesn't yet support QMF2 style Query Subscriptions.
* Console supports Agent discovery via findAgent() method, which can
support partial matches,
which is useful because the full Agent name has a UUID
representing the "instance" so it's hard
to know the full name.
* QmfQuery supports regex matching.
* Supports QMF2 WorkItem Event model and in addition supports an
alternative QmfEventListener
Event model, which is rather more like the JMS MessageListener model.
Example Tools Provided:
* ConnectionAudit: Audits connections to one or more Qpid message
brokers against a whitelist.
* ConnectionLogger: A QMF2 class used to provide information about
connections made to a broker.
* QpidConfig: QpidConfig is a fairly "literal" Java port of the
python qpid-config tool. Uses pure
QMF2 for adding/deleting queues, exchanges & bindings this
provides useful illustration of how
to do these things using the ManagementAgent method calls.
* QpidCtrl: A tool to allow QMF2 methods to be invoked from the
command line.
* QpidPrintEvents: Collect and print events from one or more Qpid
message brokers.
* QpidQueueStats: Collect and print queue statistics. This is a
rewrite of the Python version and
illustrates the use of QuerySubscriptions (via the Console side
emulation)to subscribe to
objects on the ManagementAgent.
* QueueFuse: QueueFuse provides protection to message producers from
consumers who can't consume
messages fast enough.
A couple of utility classes are provided too:
* GetOpt: Is an implementation of the Python GetOpt, which makes
command line parsing a bit less of a pain.
* ConnectionHelper: Provides support for a variety of different URL
formats. As it happens the C++ AMQP URL, the Java ConnectionURL
and the Python tool "BrokerURL" are all slightly different.
ConnectionHelper tries to support them all as well as a prototype
for the proposed new AMQP URL
https://cwiki.apache.org/qpid/url-format-proposal.html. It's
obviously still possible to create JMS Connections for QMF2 via
JNDI, but ConnectionHelper makes it a bit easier for command line
based tools where we may want to connect to lots of different
broker instances.
I hope that this is useful, I certainly find it useful as I need to do a
lot with broker management info.
It'll be really nice if this makes it into an official Qpid release at
some point.
Enjoy!!
Best Regards,
Frase
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: New Feature Announce - Java QMF2 API Implementation.
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Hi Ted,
That's great - I must admit I was getting a bit paranoid as I've not had
any feedback on this.
I think that it is quite a way ahead of the C++ QMF2 stuff, unless
things have changed recently the C++ QMF2 stuff didn't seem to follow
the QMF2 API at all, though clearly it was using the QMF2 protocol.
I'd tend to agree with yourself and the others that it's a good idea to
separate QMF from the Qpid namespace and consider QMF to be a separate
and layered capability that adds value to Qpid. That's exactly the case
with the Java API implementation, which is really to all intents and
purposes just a JMS application.
My one reservation is that if QMF got separated there may be a danger
that it could become a bit of a second class citizen, I'd be very keen
to avoid that happening. As it happens one of the reasons I started down
the path of the Java QMF2 API implementation was because the original
Java QMF stuff seemed very broken (and it was mentioned that it was not
supported/maintained and up for removal from the code base). Given my
experience with the QMF2 stuff I suspect where the original Java QMF
stuff is most likely to be broken is in the String handling. I've
mentioned in several threads lots of nasty interoperability issues due
to C++ binary strings (as opposed to utf8 encoded strings), which get
mapped into byte[]. I've put in a lot of defensive coding to cater for this.
Another thing that my implementation added to the QMF2 API was
overloaded methods on addConnection() so I can add additional
information to the address that gets created e.g.
_console.addConnection(connection, " ; {link:
{name:'big-payload-console', x-declare: {arguments: {'qpid.policy_type':
ring, 'qpid.max_size': 500000000}}}}");
This adds to the generality of QMF2, the idea here is that the QMF2
Agent and Console simply become a message based service invocation
framework, with all of the Agent/Method etc. discovery built in. Using
the extended addConnection() I can send large payloads. I can also set
explicit names so, for example if I had a service that may take a very
long time to process I can actually set it up so responses are delivered
down a named persistent queue so in theory my console could go away and
come back later to retrieve the asynchronous response.
On a (related) aside I've also written a patch for the broker to support
QMF2 query requests for types other than OBJECT see:
https://issues.apache.org/jira/browse/QPID-3696
When I was working on this patch it uncovered a couple of bugs in the
QMF2 API that I'm in the process of fixing. In actual fact they relate
to a bit of defensive coding that needed to be added where if a schema
Map got returned without a _properties or _methods key I got a null
pointer exception. In theory this shouldn't happen as they are mandatory
properties but I got bitten as I was putting together the broker patch
so it's right to add some checks. My actual finished broker patch
doesn't cause this problem as I've made sure that I included all of the
mandatory properties.
One other issue that I've uncovered is that if I do a getObjects() where
there are lots of objects (I think > 100) then the broker actually sends
multiple messages with everyone but the last makred with a "partial"
header. I only noticed this when I was writing the query patch so I need
to add support for this to the QMF2 API. It's actually what I'm working
on at the moment so I'll update the Jira when I've added this stuff in.
It would be good if you could have a play with this stuff, I've been
working on it most weekends since June so there's quite a lot of it. I
think that I've covered most of the obvious gotchas, but clearly with
this much code there are bound to be bugs.
Cheers,
Frase
On 04/01/12 13:53, Ted Ross wrote:
> Frase,
>
> Sorry for my delayed response to this. I've fallen a little behind on
> the users list.
>
> Thank you for this contribution. From your description, it looks as
> though it's a little ahead of the C++ counterpart! A Java
> implementation of QMF2 is very desirable and valuable so it should
> find its way into the code base so it can become part of the released
> software.
>
> I'm going to send an email to dev and users later today to propose
> that we separate QMF from the Qpid namespace (i.e. consider QMF to be
> a separate and layered capability that adds value to Qpid) and it
> looks like your Java implementation should be included as well.
>
> -Ted
>
> On 12/11/2011 10:52 AM, Fraser Adams wrote:
>> Hello All,
>> By way of giving something back to the Qpid community I've (finally)
>> got my implementation of the QMF2 API (which for those not already
>> familiar is described here
>> https://cwiki.apache.org/qpid/qmfv2-api-proposal.html) into a form
>> that is hopefully generally usable.
>>
>> It's taken a bit longer than I'd have liked, but I've spent the time
>> adding some example tools that are based on the API. Hopefully these
>> tools are useful, at the very least they provide a fair illustration
>> of the QMF2 API including such things as method invocation and Query
>> Subscription.
>>
>> There's quite a lot of if (just short of 100 classes including tests
>> and tools) so I've zipped it up and attached it to a Jira.
>>
>> https://issues.apache.org/jira/browse/QPID-3675
>>
>> Note that there are some quirks, especially if you use a version of
>> Qpid < 0.12. Older versions require a patch to allow the app-id
>> property name to be set. The patch is included in the distro and the
>> shell scripts that launch the tests and tools add the patch jar to
>> the front of the class path. I've tested with Qpid 0.8, 0.10 and 0.12.
>>
>> Note too that some of the features won't work with Qpid 0.8. In
>> particular the QMF2 create and delete methods weren't added until
>> Qpid 0.10 so the add queue/add binding etc. features of QpidConfig
>> won't work with Qpid 0.8, similarly for QpidCtrl.
>>
>> The code contains a fair amount of comment and JavaDoc so hopefully
>> is should be fairly easy to use.
>>
>> I certainly can't guarantee it'll be bug free :-D but I've tried
>> reasonably hard to break things, so at the very least it should be
>> robust enough to be usable. ConnectionAudit and QpidQueueStats seem
>> to reconnect and survive broker restarts pretty well.
>>
>> This is the first release of a QMF2 API Implementation for Java.
>>
>> Features:
>>
>> * Full Implementation of QMF2 Console, Agent and AgentExternal.
>> * Supports QMF2 Query Subscriptions on Agent/AgentExternal
>> implementations.
>> * Emulates QMF2 Query Subscriptions on the Console side for the
>> broker ManagementAgent by
>> intercepting _data indications and filtering against QmfQuery.
>> This is necessary as the
>> ManagemetAgent doesn't yet support QMF2 style Query Subscriptions.
>> * Console supports Agent discovery via findAgent() method, which can
>> support partial matches,
>> which is useful because the full Agent name has a UUID
>> representing the "instance" so it's hard
>> to know the full name.
>> * QmfQuery supports regex matching.
>> * Supports QMF2 WorkItem Event model and in addition supports an
>> alternative QmfEventListener
>> Event model, which is rather more like the JMS MessageListener
>> model.
>>
>> Example Tools Provided:
>>
>> * ConnectionAudit: Audits connections to one or more Qpid message
>> brokers against a whitelist.
>> * ConnectionLogger: A QMF2 class used to provide information about
>> connections made to a broker.
>> * QpidConfig: QpidConfig is a fairly "literal" Java port of the
>> python qpid-config tool. Uses pure
>> QMF2 for adding/deleting queues, exchanges & bindings this
>> provides useful illustration of how
>> to do these things using the ManagementAgent method calls.
>> * QpidCtrl: A tool to allow QMF2 methods to be invoked from the
>> command line.
>> * QpidPrintEvents: Collect and print events from one or more Qpid
>> message brokers.
>> * QpidQueueStats: Collect and print queue statistics. This is a
>> rewrite of the Python version and
>> illustrates the use of QuerySubscriptions (via the Console side
>> emulation)to subscribe to
>> objects on the ManagementAgent.
>> * QueueFuse: QueueFuse provides protection to message producers from
>> consumers who can't consume
>> messages fast enough.
>>
>> A couple of utility classes are provided too:
>>
>> * GetOpt: Is an implementation of the Python GetOpt, which makes
>> command line parsing a bit less of a pain.
>> * ConnectionHelper: Provides support for a variety of different URL
>> formats. As it happens the C++ AMQP URL, the Java ConnectionURL
>> and the Python tool "BrokerURL" are all slightly different.
>> ConnectionHelper tries to support them all as well as a prototype
>> for the proposed new AMQP URL
>> https://cwiki.apache.org/qpid/url-format-proposal.html. It's
>> obviously still possible to create JMS Connections for QMF2 via
>> JNDI, but ConnectionHelper makes it a bit easier for command line
>> based tools where we may want to connect to lots of different
>> broker instances.
>>
>>
>> I hope that this is useful, I certainly find it useful as I need to
>> do a lot with broker management info.
>>
>> It'll be really nice if this makes it into an official Qpid release
>> at some point.
>>
>> Enjoy!!
>> Best Regards,
>> Frase
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: New Feature Announce - Java QMF2 API Implementation.
Posted by Ted Ross <tr...@redhat.com>.
Frase,
Sorry for my delayed response to this. I've fallen a little behind on
the users list.
Thank you for this contribution. From your description, it looks as
though it's a little ahead of the C++ counterpart! A Java
implementation of QMF2 is very desirable and valuable so it should find
its way into the code base so it can become part of the released software.
I'm going to send an email to dev and users later today to propose that
we separate QMF from the Qpid namespace (i.e. consider QMF to be a
separate and layered capability that adds value to Qpid) and it looks
like your Java implementation should be included as well.
-Ted
On 12/11/2011 10:52 AM, Fraser Adams wrote:
> Hello All,
> By way of giving something back to the Qpid community I've (finally)
> got my implementation of the QMF2 API (which for those not already
> familiar is described here
> https://cwiki.apache.org/qpid/qmfv2-api-proposal.html) into a form
> that is hopefully generally usable.
>
> It's taken a bit longer than I'd have liked, but I've spent the time
> adding some example tools that are based on the API. Hopefully these
> tools are useful, at the very least they provide a fair illustration
> of the QMF2 API including such things as method invocation and Query
> Subscription.
>
> There's quite a lot of if (just short of 100 classes including tests
> and tools) so I've zipped it up and attached it to a Jira.
>
> https://issues.apache.org/jira/browse/QPID-3675
>
> Note that there are some quirks, especially if you use a version of
> Qpid < 0.12. Older versions require a patch to allow the app-id
> property name to be set. The patch is included in the distro and the
> shell scripts that launch the tests and tools add the patch jar to the
> front of the class path. I've tested with Qpid 0.8, 0.10 and 0.12.
>
> Note too that some of the features won't work with Qpid 0.8. In
> particular the QMF2 create and delete methods weren't added until Qpid
> 0.10 so the add queue/add binding etc. features of QpidConfig won't
> work with Qpid 0.8, similarly for QpidCtrl.
>
> The code contains a fair amount of comment and JavaDoc so hopefully is
> should be fairly easy to use.
>
> I certainly can't guarantee it'll be bug free :-D but I've tried
> reasonably hard to break things, so at the very least it should be
> robust enough to be usable. ConnectionAudit and QpidQueueStats seem to
> reconnect and survive broker restarts pretty well.
>
> This is the first release of a QMF2 API Implementation for Java.
>
> Features:
>
> * Full Implementation of QMF2 Console, Agent and AgentExternal.
> * Supports QMF2 Query Subscriptions on Agent/AgentExternal
> implementations.
> * Emulates QMF2 Query Subscriptions on the Console side for the
> broker ManagementAgent by
> intercepting _data indications and filtering against QmfQuery.
> This is necessary as the
> ManagemetAgent doesn't yet support QMF2 style Query Subscriptions.
> * Console supports Agent discovery via findAgent() method, which can
> support partial matches,
> which is useful because the full Agent name has a UUID
> representing the "instance" so it's hard
> to know the full name.
> * QmfQuery supports regex matching.
> * Supports QMF2 WorkItem Event model and in addition supports an
> alternative QmfEventListener
> Event model, which is rather more like the JMS MessageListener
> model.
>
> Example Tools Provided:
>
> * ConnectionAudit: Audits connections to one or more Qpid message
> brokers against a whitelist.
> * ConnectionLogger: A QMF2 class used to provide information about
> connections made to a broker.
> * QpidConfig: QpidConfig is a fairly "literal" Java port of the
> python qpid-config tool. Uses pure
> QMF2 for adding/deleting queues, exchanges & bindings this
> provides useful illustration of how
> to do these things using the ManagementAgent method calls.
> * QpidCtrl: A tool to allow QMF2 methods to be invoked from the
> command line.
> * QpidPrintEvents: Collect and print events from one or more Qpid
> message brokers.
> * QpidQueueStats: Collect and print queue statistics. This is a
> rewrite of the Python version and
> illustrates the use of QuerySubscriptions (via the Console side
> emulation)to subscribe to
> objects on the ManagementAgent.
> * QueueFuse: QueueFuse provides protection to message producers from
> consumers who can't consume
> messages fast enough.
>
> A couple of utility classes are provided too:
>
> * GetOpt: Is an implementation of the Python GetOpt, which makes
> command line parsing a bit less of a pain.
> * ConnectionHelper: Provides support for a variety of different URL
> formats. As it happens the C++ AMQP URL, the Java ConnectionURL
> and the Python tool "BrokerURL" are all slightly different.
> ConnectionHelper tries to support them all as well as a prototype
> for the proposed new AMQP URL
> https://cwiki.apache.org/qpid/url-format-proposal.html. It's
> obviously still possible to create JMS Connections for QMF2 via
> JNDI, but ConnectionHelper makes it a bit easier for command line
> based tools where we may want to connect to lots of different
> broker instances.
>
>
> I hope that this is useful, I certainly find it useful as I need to do
> a lot with broker management info.
>
> It'll be really nice if this makes it into an official Qpid release at
> some point.
>
> Enjoy!!
> Best Regards,
> Frase
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org