You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Fraser Adams <fr...@blueyonder.co.uk> on 2014/01/24 00:54:11 UTC
QMF REST API - was Re: How to use lifetime policies
Hey Davin,
Robbie has hit the nail on the head. One somewhat unfortunate thing that
you probably ought to be aware of is that the management and monitoring
for the Qpid C++ and Java Brokers got somewhat erm "fragmented" for
various reasons back in the mists of time.
The C++ Broker uses Qpid Management Framework (QMF), but that has had an
interesting evolution and at some point in history QMFv1 migrated to
QMFv2 which caused issues for the Java Broker and they ended up going
down a different path.
I primarily use the C++ broker, but conversely my team mainly use Java,
so I started out doing a Java implementation of the QMF2 API without
knowing much of that background. There was a bit of comedy 'cause the
Qpid Java guys didn't take a lot of notice of what I was doing 'cause it
was QMF and the C++ guys didn't really notice 'cause it was Java :-D I
was a bit in no mans land really :-/
It wasn't really until I put the GUI together that anyone really noticed
and the penny dropped.
So as Robbie says the REST API that I put together is *actually* a REST
binding for QMF2 and (apart from being a REST API) doesn't bear too much
relationship to the Java Broker's REST interface (for most of history we
were largely unaware of what each other was up to).
It's a far from ideal position and Rob, Gordon, Robbie and I have
chatted a fair bit about the need to get a more cohesive view.
Ultimately the right answer will be to adopt the AMQP Management
Specification across the board, though that's likely to be some way off
(I certainly need to find some time to properly get my head around it).
By way of "making a stand for the unity of Qpid Brokers :-)" I put
together a QMF2 plugin for the Java Broker, it was one of those things
that one does "on principle" - really it was as much as anything a
statement to say "really guys it *is* actually possible to provide a
cohesive view". One thing to bear in mind with the Java Broker QMF
plugin is that I've tried fairly hard to map as much as possible from
the Java Broker internal Management Model to QMF Management Objects, but
there are differences in the Management Models, so there are certainly
more itty bitty things that can be controlled by the Java Broker's
native management interface than via QMF (mainly 'cause Robbie keeps
adding features ;->) but OTOH pretty much anything you can do with
qpid-config will work with both the C++ and Java Brokers.
So to get back to the QMF REST API, the best place to look is the
JavaDoc (yes really, there is actually a ton of JavaDoc for this stuff)
start in <qpid>/tools/src/java/docs/api/index.html
Then look for package org.apache.qpid.restapi and class QpidServer
To get you started if you do (assuming the same host and port as below):
http://127.0.0.1:8080/qpid/connection/default/console/objects/queue
You will retrieve a list of all of the QMF Queue Management Objects in
the form of a JSON list (you can do it in a browser too)
http://127.0.0.1:8080/qpid/connection/default/console/objects/exchange
will retrieve the exchanges
http://127.0.0.1:8080/qpid/connection/default/console/objects/connection
will retrieve the connections
and so on.
the <default> bit is a "convenience", so this is a "true" REST API that
mirrors the QMF API pretty closely so you really ought to first PUT a
QMF Console Connection before you do anything else, but of course PUT is
a pain from a browser, so if you specify "default" for the connection
part of the URI the API transparently creates (or uses an already
created) default QMF Console instance.
The GUI actually has a JavaScript class that provides a JavaScript
implementation of the QMF2 API, which is backed by the REST API, so the
GUI is generally abstracted from the REST API, there's a factory method
for that which does the HTTP PUT "under the hood"
You can also use POST in order to invoke QMF methods (for
adding/deleting queues etc.).
To be honest I'm a bit rusty - I tend to use the GUI or at worse the
JavaScript wrapper which abstracts the detail,
and I've not done it "the hard way" for the best part of 15 months, so
the following is pulled out from the code.
The POST syntax is:
POST: <host>:<port>/qpid/connection/<name>/object/<ObjectId>
HTTP body: {"_method_name":<method>,"_arguments":<inArgs>}
<method>: A string containing the QMF2 method name e.g. "getLogLevel", "setLogLevel", "create", "delete".
<inArgs>: A JSON string containing the method arguments e.g. {"level":"debug+:Broker"} for setLogLevel.
HTTP response: A JSON string containing the response e.g. {"level":"notice+"} for getLogLevel (may be empty).
This method invokes the QMF2 method <method> with arguments <inArgs> on the object <ObjectId>
And it works like QMF, so what you'd first need to do is to recover the Broker Management Object
http://127.0.0.1:8080/qpid/connection/default/console/objects/broker
You need to do this because the majority of the QMF CRUD methods are applied on the Broker Management Object,
having done this you'd need to do a POST as above (using the ObjectId from your own Broker Object)
POST http://127.0.0.1:8080/qpid/connection/default/object/@0@org.apache.qpid.broker:broker:amqp-broker
And in the HTTP body you need a JSON string.
To Add a queue I *think* that the JSON body goes something like this (for your davin_test2 queue):
{
"_method_name":"create",
"_arguments": {"type": "queue", "name": "davin_test2", "properties": {"durable": true}}
}
Clearly this isn't something that one would *generally* do "by hand".
I've just done:
curl -X POST -u admin:admin -d '{"_method_name":"create","_arguments": {"type": "queue", "name": "davin_test2", "properties": {"durable": true}}}'
http://127.0.0.1:8080/qpid/connection/default/object/@0@org.apache.qpid.broker:broker:amqp-broker
On my system and it created a queue called "davin_test2", so I think what I've just said is accurate, yay!!
The available properties for Queues and exchanges isn't brilliantly documented in the QMF Management Schema
(or anywhere really) and I kind of "reverse engineered" them, looking through the GUI code (qmf-ui.js in qmfui.AddQueue method)
I can see:
qpid.max_size
qpid.max_count
qpid.flow_stop_size
qpid.flow_stop_count
qpid.flow_resume_size
qpid.flow_resume_count
durable
qpid.file_size
qpid.file_count
qpid.policy_type
qpid.last_value_queue
qpid.last_value_queue_no_browse
qpid.queue_event_generation
alternate-exchange
If you look in the qpid-config code I think you'd come across these too.
I've *probably* just made your head explode, so I'd better stop there, *hopefully* this is helpful and useful background.
Best regards,
Frase
P.S. Glad that you like the QMF GUI - it looks pretty nice on an iOS device, it uses CSS3 animations so the transitions are GPU accelerated.
On 23/01/14 19:42, Robbie Gemmell wrote:
> On 23 January 2014 19:27, Shearer, Davin <ds...@novetta.com> wrote:
>
>>
>> [davin@laptop24 qpid]$ curl -X GET -u admin:admin
>> http://127.0.0.1:8080/rest/queue/davin_test
>> 404 Not Found: File /rest/queue/davin_test not found.[davin@laptop24qpid]$
>>
>> Darn, that didn't work... can we put new queues?
>>
>> [davin@laptop24 qpid]$ curl -X PUT -u admin:admin -d '{"durable":true}'
>> http://127.0.0.1:8080/rest/queue/default/davin_test2
>> 405 Bad Method.[davin@laptop24 qpid]$
>>
>> What am I missing?
>>
> Those URLs look a lot like ones that would be used against the REST API
> from the Java brokers management-http plugin. The URLs/payloads used by the
> 'QMF REST API' from the tools Fraser put together aren't likely to be that
> similar to them.
>
> I'll now have to defer to someone who can tell you what they would be
> like...
>
> Robbie
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: QMF REST API - was Re: How to use lifetime policies
Posted by "Shearer, Davin" <ds...@novetta.com>.
By having this transport and messaging layer, I can swap out the broker
management implementations (hopefully) without changing the API. If my
customers demand that I support a different AMQP 1.0 vendor, then I can
load in a different management module. If they have a REST API, then I
take on no additional dependencies. If I could have managed to link
libqpid* statically, it would have helped. Once the management spec
arrives and there is a proton-like library implementing that spec, then
I'll have my final solution.
I've learned a bunch from you guys over the past couple of weeks. Thanks
so much for the help!
Re: QMF REST API - was Re: How to use lifetime policies
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Re: "I like the decoupling that REST provides as it just takes a library
that speaks HTTP. I don't have to link against a vendor-specific
management library nor do I have to carry through their dependencies. "
What you say about REST is indeed true and is a good selling point, but
it's worth bearing in mind that it's *also* true of the approach
illustrated in the examples from Gordon - that approach is using the QMF
*protocol* and in terms of dependencies it is simply using pure AMQP as
a transport passing Map Messages in a request/response pattern to
specific AMQP Destinations, that's arguably no more coupled nor vendor
specific than say using the REST API, even the Java QMF stuff isn't
really coupled as it's implemented as pure JMS. In actual fact it's the
QMF aspects that are vendor specific - QMF is specific to the Qpid C++
broker.
I certainly don't want to dissuade the approach that you're taking, but
you should probably be armed with all the facts - sure you're not
dependent on a library, but as I say above nor are you really dependent
on a library for some of the other approaches (though Gordon's examples
are C++ using qpid::mess
I like the decoupling that REST provides as it just
takes a library that speaks HTTP. I don't have to link against a
vendor-specific management library nor do I have to carry through their
dependencies.
aging) I've not tried raw QMF via Proton Messenger, it might well be
possible though it currently seems to have quite a few limitations wrt.
topic subscriptions (and other filter types).
As I say ultimately QMF is actually your biggest vendor specific
dependency, so that's something that you should keep in the back of your
mind going forward. I expect QMF to be around for quite some time, but
there is certainly a desire to move towards the AMQP 1.0 Management
Specification over time. As we move forward on that I'm sure that there
will be a concerted effort to avoid breaking existing QMF things (I'm
certainly intending to try my best to avoid that) and as I say it's
likely to be some time before traction builds in that direction, but
it's certainly something that you should be cognoscente of if you are
building a mission critical system - put a comment in your code to track
AMQP 1.0 Management developments on this list at the very least.
Hope that this stuff is useful,
Cheers,
Frase
On 28/01/14 19:18, Shearer, Davin wrote:
> On Tue, Jan 28, 2014 at 1:39 PM, Fraser Adams <fraser.adams@blueyonder.co.uk
>> wrote:
>> Glad that I was able to help out.
>> I'm a little bit curious why you're choosing to go with the REST interface
>> directly? What language are you programming in.
>>
> I am programming this piece in C (messaging and transport layer of a larger
> system). I am using proton for the AMQP layer and using the REST API for
> the broker management. I like the decoupling that REST provides as it just
> takes a library that speaks HTTP. I don't have to link against a
> vendor-specific management library nor do I have to carry through their
> dependencies. I don't want the consumers of the library (be it additional
> middle-ware layers or the application-layer code) to be concerned about the
> messaging and transport details, just that we moving data reliably from one
> end-point to another over a flexible topology.
>
>
>> Not that I've got anything against the REST API, but I mainly put it in
>> place to be able to talk QMF from a browser environment.
>>
> It should work for my needs as well. It took longer than I'd hoped to
> package a "reference" broker, but I've managed it.
>
>
>> I've attached a couple of examples put together by Gordon Sim to
>> illustrate this in C++ the queue_depth is self explanatory, the one marked
>> example.cpp does something similar to qpid-route - not exactly what you
>> want for adding a queue, but it illustrates invoking QMF2 methods via the
>> raw QMF2 protocol and you can use that in conjunction with what I've shown
>> below on the REST API
>>
> I think your REST API will be sufficient provided I request a dynamic queue
> with delete on close via REST. If that isn't possible then it could be the
> case that there will be abandoned reply queues lingering in the broker if a
> client creates one, then dies without removing it. I will review the code
> since more information usually leads to better decisions.
>
>
>> It is quite fun doing this stuff via curl, but it's pretty hardcore :-)
>> I'd *probably* take easier options in Python/Java/C++/JavaScript unless I
>> wanted to prove a point (clearly REST is handy if you want to mash up info
>> from lots of different sources in a web app).
>>
> The commandline curl stuff is to understand the API, get my URLs right and
> to test the behaviour before writing it up in C using libcurl.
>
>
>> BTW in case you are curious the reason that the URI's are relatively
>> complex e.g.
>> GET http://127.0.0.1:8080/qpid/connection/default/console/objects/queuerather than say
>> GET http://127.0.0.1:8080/qpid/console/objects/queue
>>
>> Is because it's perfectly possible to PUT an arbitrary number of QMF
>> Console connections and access them independently so the GUI can talk to
>> multiple discrete Consoles and I can have multiple browser instances served
>> by the same REST API instance.
>>
> I think the flexibility that provides is worth the price in complexity. I
> only have to get it right once, then it's in my library and I'll build on
> top of that. :-)
>
> I hope that helpful and that it made sense...
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: QMF REST API - was Re: How to use lifetime policies
Posted by "Shearer, Davin" <ds...@novetta.com>.
On Tue, Jan 28, 2014 at 1:39 PM, Fraser Adams <fraser.adams@blueyonder.co.uk
> wrote:
> Glad that I was able to help out.
> I'm a little bit curious why you're choosing to go with the REST interface
> directly? What language are you programming in.
>
I am programming this piece in C (messaging and transport layer of a larger
system). I am using proton for the AMQP layer and using the REST API for
the broker management. I like the decoupling that REST provides as it just
takes a library that speaks HTTP. I don't have to link against a
vendor-specific management library nor do I have to carry through their
dependencies. I don't want the consumers of the library (be it additional
middle-ware layers or the application-layer code) to be concerned about the
messaging and transport details, just that we moving data reliably from one
end-point to another over a flexible topology.
> Not that I've got anything against the REST API, but I mainly put it in
> place to be able to talk QMF from a browser environment.
>
It should work for my needs as well. It took longer than I'd hoped to
package a "reference" broker, but I've managed it.
> I've attached a couple of examples put together by Gordon Sim to
> illustrate this in C++ the queue_depth is self explanatory, the one marked
> example.cpp does something similar to qpid-route - not exactly what you
> want for adding a queue, but it illustrates invoking QMF2 methods via the
> raw QMF2 protocol and you can use that in conjunction with what I've shown
> below on the REST API
>
I think your REST API will be sufficient provided I request a dynamic queue
with delete on close via REST. If that isn't possible then it could be the
case that there will be abandoned reply queues lingering in the broker if a
client creates one, then dies without removing it. I will review the code
since more information usually leads to better decisions.
> It is quite fun doing this stuff via curl, but it's pretty hardcore :-)
> I'd *probably* take easier options in Python/Java/C++/JavaScript unless I
> wanted to prove a point (clearly REST is handy if you want to mash up info
> from lots of different sources in a web app).
>
The commandline curl stuff is to understand the API, get my URLs right and
to test the behaviour before writing it up in C using libcurl.
> BTW in case you are curious the reason that the URI's are relatively
> complex e.g.
> GET http://127.0.0.1:8080/qpid/connection/default/console/objects/queuerather than say
> GET http://127.0.0.1:8080/qpid/console/objects/queue
>
> Is because it's perfectly possible to PUT an arbitrary number of QMF
> Console connections and access them independently so the GUI can talk to
> multiple discrete Consoles and I can have multiple browser instances served
> by the same REST API instance.
>
I think the flexibility that provides is worth the price in complexity. I
only have to get it right once, then it's in my library and I'll build on
top of that. :-)
I hope that helpful and that it made sense...
--
Davin Shearer
Re: QMF REST API - was Re: How to use lifetime policies
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Glad that I was able to help out.
I'm a little bit curious why you're choosing to go with the REST
interface directly? What language are you programming in.
Not that I've got anything against the REST API, but I mainly put it in
place to be able to talk QMF from a browser environment.
If you are programming in Java you'd probably find it easier to use the
Java QMF API directly, if you're doing python then use the API that
qpid-config etc. use, from C++ it's a little more awkward but not too
hard to send QMF2 protocol requests out (it's ultimately just a bunch of
Map Messages) I've attached a couple of examples put together by Gordon
Sim to illustrate this in C++ the queue_depth is self explanatory, the
one marked example.cpp does something similar to qpid-route - not
exactly what you want for adding a queue, but it illustrates invoking
QMF2 methods via the raw QMF2 protocol and you can use that in
conjunction with what I've shown below on the REST API (this bit
{
"_method_name":"create",
"_arguments": {"type": "queue", "name": "davin_test2", "properties":
{"durable": true}}
}
)
It is quite fun doing this stuff via curl, but it's pretty hardcore :-)
I'd *probably* take easier options in Python/Java/C++/JavaScript unless
I wanted to prove a point (clearly REST is handy if you want to mash up
info from lots of different sources in a web app).
BTW in case you are curious the reason that the URI's are relatively
complex e.g.
GET http://127.0.0.1:8080/qpid/connection/default/console/objects/queue
rather than say
GET http://127.0.0.1:8080/qpid/console/objects/queue
Is because it's perfectly possible to PUT an arbitrary number of QMF
Console connections and access them independently so the GUI can talk to
multiple discrete Consoles and I can have multiple browser instances
served by the same REST API instance.
Cheers,
Frase
On 27/01/14 22:06, Shearer, Davin wrote:
> Thanks Frase,
>
> I read and followed along through your explanation and was able to browse
> various objects and finally create a queue via the REST API backed by the
> C++ broker using only curl (yay!). Hopefully I can use the interface to
> create one dynamically with the delete on close attribute. I'll give that
> a go tomorrow!
>
>
> On Thu, Jan 23, 2014 at 6:54 PM, Fraser Adams <fraser.adams@blueyonder.co.uk
>> wrote:
>> Hey Davin,
>> Robbie has hit the nail on the head. One somewhat unfortunate thing that
>> you probably ought to be aware of is that the management and monitoring for
>> the Qpid C++ and Java Brokers got somewhat erm "fragmented" for various
>> reasons back in the mists of time.
>>
>> The C++ Broker uses Qpid Management Framework (QMF), but that has had an
>> interesting evolution and at some point in history QMFv1 migrated to QMFv2
>> which caused issues for the Java Broker and they ended up going down a
>> different path.
>>
>> I primarily use the C++ broker, but conversely my team mainly use Java, so
>> I started out doing a Java implementation of the QMF2 API without knowing
>> much of that background. There was a bit of comedy 'cause the Qpid Java
>> guys didn't take a lot of notice of what I was doing 'cause it was QMF and
>> the C++ guys didn't really notice 'cause it was Java :-D I was a bit in no
>> mans land really :-/
>>
>> It wasn't really until I put the GUI together that anyone really noticed
>> and the penny dropped.
>>
>> So as Robbie says the REST API that I put together is *actually* a REST
>> binding for QMF2 and (apart from being a REST API) doesn't bear too much
>> relationship to the Java Broker's REST interface (for most of history we
>> were largely unaware of what each other was up to).
>>
>> It's a far from ideal position and Rob, Gordon, Robbie and I have chatted
>> a fair bit about the need to get a more cohesive view. Ultimately the right
>> answer will be to adopt the AMQP Management Specification across the board,
>> though that's likely to be some way off (I certainly need to find some time
>> to properly get my head around it).
>>
>> By way of "making a stand for the unity of Qpid Brokers :-)" I put
>> together a QMF2 plugin for the Java Broker, it was one of those things that
>> one does "on principle" - really it was as much as anything a statement to
>> say "really guys it *is* actually possible to provide a cohesive view". One
>> thing to bear in mind with the Java Broker QMF plugin is that I've tried
>> fairly hard to map as much as possible from the Java Broker internal
>> Management Model to QMF Management Objects, but there are differences in
>> the Management Models, so there are certainly more itty bitty things that
>> can be controlled by the Java Broker's native management interface than via
>> QMF (mainly 'cause Robbie keeps adding features ;->) but OTOH pretty much
>> anything you can do with qpid-config will work with both the C++ and Java
>> Brokers.
>>
>> So to get back to the QMF REST API, the best place to look is the JavaDoc
>> (yes really, there is actually a ton of JavaDoc for this stuff)
>> start in <qpid>/tools/src/java/docs/api/index.html
>> Then look for package org.apache.qpid.restapi and class QpidServer
>>
>> To get you started if you do (assuming the same host and port as below):
>> http://127.0.0.1:8080/qpid/connection/default/console/objects/queue
>>
>> You will retrieve a list of all of the QMF Queue Management Objects in the
>> form of a JSON list (you can do it in a browser too)
>> http://127.0.0.1:8080/qpid/connection/default/console/objects/exchangewill retrieve the exchanges
>> http://127.0.0.1:8080/qpid/connection/default/console/objects/connectionwill retrieve the connections
>> and so on.
>>
>> the <default> bit is a "convenience", so this is a "true" REST API that
>> mirrors the QMF API pretty closely so you really ought to first PUT a QMF
>> Console Connection before you do anything else, but of course PUT is a pain
>> from a browser, so if you specify "default" for the connection part of the
>> URI the API transparently creates (or uses an already created) default QMF
>> Console instance.
>>
>> The GUI actually has a JavaScript class that provides a JavaScript
>> implementation of the QMF2 API, which is backed by the REST API, so the GUI
>> is generally abstracted from the REST API, there's a factory method for
>> that which does the HTTP PUT "under the hood"
>>
>> You can also use POST in order to invoke QMF methods (for adding/deleting
>> queues etc.).
>> To be honest I'm a bit rusty - I tend to use the GUI or at worse the
>> JavaScript wrapper which abstracts the detail,
>> and I've not done it "the hard way" for the best part of 15 months, so the
>> following is pulled out from the code.
>>
>> The POST syntax is:
>>
>> POST: <host>:<port>/qpid/connection/<name>/object/<ObjectId>
>> HTTP body: {"_method_name":<method>,"_arguments":<inArgs>}
>> <method>: A string containing the QMF2 method name e.g.
>> "getLogLevel", "setLogLevel", "create", "delete".
>> <inArgs>: A JSON string containing the method arguments e.g.
>> {"level":"debug+:Broker"} for setLogLevel.
>> HTTP response: A JSON string containing the response e.g.
>> {"level":"notice+"} for getLogLevel (may be empty).
>>
>> This method invokes the QMF2 method <method> with arguments <inArgs>
>> on the object <ObjectId>
>>
>> And it works like QMF, so what you'd first need to do is to recover the
>> Broker Management Object
>> http://127.0.0.1:8080/qpid/connection/default/console/objects/broker
>>
>> You need to do this because the majority of the QMF CRUD methods are
>> applied on the Broker Management Object,
>> having done this you'd need to do a POST as above (using the ObjectId from
>> your own Broker Object)
>> POST http://127.0.0.1:8080/qpid/connection/default/object/@0@
>> org.apache.qpid.broker:broker:amqp-broker
>> And in the HTTP body you need a JSON string.
>>
>> To Add a queue I *think* that the JSON body goes something like this (for
>> your davin_test2 queue):
>>
>> {
>> "_method_name":"create",
>> "_arguments": {"type": "queue", "name": "davin_test2", "properties":
>> {"durable": true}}
>> }
>>
>> Clearly this isn't something that one would *generally* do "by hand".
>>
>> I've just done:
>>
>> curl -X POST -u admin:admin -d '{"_method_name":"create","_arguments":
>> {"type": "queue", "name": "davin_test2", "properties": {"durable": true}}}'
>> http://127.0.0.1:8080/qpid/connection/default/object/@0@
>> org.apache.qpid.broker:broker:amqp-broker
>>
>> On my system and it created a queue called "davin_test2", so I think what
>> I've just said is accurate, yay!!
>>
>>
>> The available properties for Queues and exchanges isn't brilliantly
>> documented in the QMF Management Schema
>> (or anywhere really) and I kind of "reverse engineered" them, looking
>> through the GUI code (qmf-ui.js in qmfui.AddQueue method)
>> I can see:
>> qpid.max_size
>> qpid.max_count
>> qpid.flow_stop_size
>> qpid.flow_stop_count
>> qpid.flow_resume_size
>> qpid.flow_resume_count
>> durable
>> qpid.file_size
>> qpid.file_count
>> qpid.policy_type
>> qpid.last_value_queue
>> qpid.last_value_queue_no_browse
>> qpid.queue_event_generation
>> alternate-exchange
>>
>> If you look in the qpid-config code I think you'd come across these too.
>>
>> I've *probably* just made your head explode, so I'd better stop there,
>> *hopefully* this is helpful and useful background.
>>
>> Best regards,
>> Frase
>>
>> P.S. Glad that you like the QMF GUI - it looks pretty nice on an iOS
>> device, it uses CSS3 animations so the transitions are GPU accelerated.
>>
>>
>> On 23/01/14 19:42, Robbie Gemmell wrote:
>>
>>> On 23 January 2014 19:27, Shearer, Davin <ds...@novetta.com> wrote:
>>>
>>>
>>>> [davin@laptop24 qpid]$ curl -X GET -u admin:admin
>>>> http://127.0.0.1:8080/rest/queue/davin_test
>>>> 404 Not Found: File /rest/queue/davin_test not found.[davin@laptop24qpid
>>>> ]$
>>>>
>>>> Darn, that didn't work... can we put new queues?
>>>>
>>>> [davin@laptop24 qpid]$ curl -X PUT -u admin:admin -d '{"durable":true}'
>>>> http://127.0.0.1:8080/rest/queue/default/davin_test2
>>>> 405 Bad Method.[davin@laptop24 qpid]$
>>>>
>>>> What am I missing?
>>>>
>>>> Those URLs look a lot like ones that would be used against the REST API
>>> from the Java brokers management-http plugin. The URLs/payloads used by
>>> the
>>> 'QMF REST API' from the tools Fraser put together aren't likely to be that
>>> similar to them.
>>>
>>> I'll now have to defer to someone who can tell you what they would be
>>> like...
>>>
>>> Robbie
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
Re: QMF REST API - was Re: How to use lifetime policies
Posted by "Shearer, Davin" <ds...@novetta.com>.
Thanks Frase,
I read and followed along through your explanation and was able to browse
various objects and finally create a queue via the REST API backed by the
C++ broker using only curl (yay!). Hopefully I can use the interface to
create one dynamically with the delete on close attribute. I'll give that
a go tomorrow!
On Thu, Jan 23, 2014 at 6:54 PM, Fraser Adams <fraser.adams@blueyonder.co.uk
> wrote:
> Hey Davin,
> Robbie has hit the nail on the head. One somewhat unfortunate thing that
> you probably ought to be aware of is that the management and monitoring for
> the Qpid C++ and Java Brokers got somewhat erm "fragmented" for various
> reasons back in the mists of time.
>
> The C++ Broker uses Qpid Management Framework (QMF), but that has had an
> interesting evolution and at some point in history QMFv1 migrated to QMFv2
> which caused issues for the Java Broker and they ended up going down a
> different path.
>
> I primarily use the C++ broker, but conversely my team mainly use Java, so
> I started out doing a Java implementation of the QMF2 API without knowing
> much of that background. There was a bit of comedy 'cause the Qpid Java
> guys didn't take a lot of notice of what I was doing 'cause it was QMF and
> the C++ guys didn't really notice 'cause it was Java :-D I was a bit in no
> mans land really :-/
>
> It wasn't really until I put the GUI together that anyone really noticed
> and the penny dropped.
>
> So as Robbie says the REST API that I put together is *actually* a REST
> binding for QMF2 and (apart from being a REST API) doesn't bear too much
> relationship to the Java Broker's REST interface (for most of history we
> were largely unaware of what each other was up to).
>
> It's a far from ideal position and Rob, Gordon, Robbie and I have chatted
> a fair bit about the need to get a more cohesive view. Ultimately the right
> answer will be to adopt the AMQP Management Specification across the board,
> though that's likely to be some way off (I certainly need to find some time
> to properly get my head around it).
>
> By way of "making a stand for the unity of Qpid Brokers :-)" I put
> together a QMF2 plugin for the Java Broker, it was one of those things that
> one does "on principle" - really it was as much as anything a statement to
> say "really guys it *is* actually possible to provide a cohesive view". One
> thing to bear in mind with the Java Broker QMF plugin is that I've tried
> fairly hard to map as much as possible from the Java Broker internal
> Management Model to QMF Management Objects, but there are differences in
> the Management Models, so there are certainly more itty bitty things that
> can be controlled by the Java Broker's native management interface than via
> QMF (mainly 'cause Robbie keeps adding features ;->) but OTOH pretty much
> anything you can do with qpid-config will work with both the C++ and Java
> Brokers.
>
> So to get back to the QMF REST API, the best place to look is the JavaDoc
> (yes really, there is actually a ton of JavaDoc for this stuff)
> start in <qpid>/tools/src/java/docs/api/index.html
> Then look for package org.apache.qpid.restapi and class QpidServer
>
> To get you started if you do (assuming the same host and port as below):
> http://127.0.0.1:8080/qpid/connection/default/console/objects/queue
>
> You will retrieve a list of all of the QMF Queue Management Objects in the
> form of a JSON list (you can do it in a browser too)
> http://127.0.0.1:8080/qpid/connection/default/console/objects/exchangewill retrieve the exchanges
> http://127.0.0.1:8080/qpid/connection/default/console/objects/connectionwill retrieve the connections
> and so on.
>
> the <default> bit is a "convenience", so this is a "true" REST API that
> mirrors the QMF API pretty closely so you really ought to first PUT a QMF
> Console Connection before you do anything else, but of course PUT is a pain
> from a browser, so if you specify "default" for the connection part of the
> URI the API transparently creates (or uses an already created) default QMF
> Console instance.
>
> The GUI actually has a JavaScript class that provides a JavaScript
> implementation of the QMF2 API, which is backed by the REST API, so the GUI
> is generally abstracted from the REST API, there's a factory method for
> that which does the HTTP PUT "under the hood"
>
> You can also use POST in order to invoke QMF methods (for adding/deleting
> queues etc.).
> To be honest I'm a bit rusty - I tend to use the GUI or at worse the
> JavaScript wrapper which abstracts the detail,
> and I've not done it "the hard way" for the best part of 15 months, so the
> following is pulled out from the code.
>
> The POST syntax is:
>
> POST: <host>:<port>/qpid/connection/<name>/object/<ObjectId>
> HTTP body: {"_method_name":<method>,"_arguments":<inArgs>}
> <method>: A string containing the QMF2 method name e.g.
> "getLogLevel", "setLogLevel", "create", "delete".
> <inArgs>: A JSON string containing the method arguments e.g.
> {"level":"debug+:Broker"} for setLogLevel.
> HTTP response: A JSON string containing the response e.g.
> {"level":"notice+"} for getLogLevel (may be empty).
>
> This method invokes the QMF2 method <method> with arguments <inArgs>
> on the object <ObjectId>
>
> And it works like QMF, so what you'd first need to do is to recover the
> Broker Management Object
> http://127.0.0.1:8080/qpid/connection/default/console/objects/broker
>
> You need to do this because the majority of the QMF CRUD methods are
> applied on the Broker Management Object,
> having done this you'd need to do a POST as above (using the ObjectId from
> your own Broker Object)
> POST http://127.0.0.1:8080/qpid/connection/default/object/@0@
> org.apache.qpid.broker:broker:amqp-broker
> And in the HTTP body you need a JSON string.
>
> To Add a queue I *think* that the JSON body goes something like this (for
> your davin_test2 queue):
>
> {
> "_method_name":"create",
> "_arguments": {"type": "queue", "name": "davin_test2", "properties":
> {"durable": true}}
> }
>
> Clearly this isn't something that one would *generally* do "by hand".
>
> I've just done:
>
> curl -X POST -u admin:admin -d '{"_method_name":"create","_arguments":
> {"type": "queue", "name": "davin_test2", "properties": {"durable": true}}}'
> http://127.0.0.1:8080/qpid/connection/default/object/@0@
> org.apache.qpid.broker:broker:amqp-broker
>
> On my system and it created a queue called "davin_test2", so I think what
> I've just said is accurate, yay!!
>
>
> The available properties for Queues and exchanges isn't brilliantly
> documented in the QMF Management Schema
> (or anywhere really) and I kind of "reverse engineered" them, looking
> through the GUI code (qmf-ui.js in qmfui.AddQueue method)
> I can see:
> qpid.max_size
> qpid.max_count
> qpid.flow_stop_size
> qpid.flow_stop_count
> qpid.flow_resume_size
> qpid.flow_resume_count
> durable
> qpid.file_size
> qpid.file_count
> qpid.policy_type
> qpid.last_value_queue
> qpid.last_value_queue_no_browse
> qpid.queue_event_generation
> alternate-exchange
>
> If you look in the qpid-config code I think you'd come across these too.
>
> I've *probably* just made your head explode, so I'd better stop there,
> *hopefully* this is helpful and useful background.
>
> Best regards,
> Frase
>
> P.S. Glad that you like the QMF GUI - it looks pretty nice on an iOS
> device, it uses CSS3 animations so the transitions are GPU accelerated.
>
>
> On 23/01/14 19:42, Robbie Gemmell wrote:
>
>> On 23 January 2014 19:27, Shearer, Davin <ds...@novetta.com> wrote:
>>
>>
>>> [davin@laptop24 qpid]$ curl -X GET -u admin:admin
>>> http://127.0.0.1:8080/rest/queue/davin_test
>>> 404 Not Found: File /rest/queue/davin_test not found.[davin@laptop24qpid
>>> ]$
>>>
>>> Darn, that didn't work... can we put new queues?
>>>
>>> [davin@laptop24 qpid]$ curl -X PUT -u admin:admin -d '{"durable":true}'
>>> http://127.0.0.1:8080/rest/queue/default/davin_test2
>>> 405 Bad Method.[davin@laptop24 qpid]$
>>>
>>> What am I missing?
>>>
>>> Those URLs look a lot like ones that would be used against the REST API
>> from the Java brokers management-http plugin. The URLs/payloads used by
>> the
>> 'QMF REST API' from the tools Fraser put together aren't likely to be that
>> similar to them.
>>
>> I'll now have to defer to someone who can tell you what they would be
>> like...
>>
>> Robbie
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
--
Davin Shearer
Re: QMF REST API - was Re: How to use lifetime policies
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Many thanks for the clarification Robbie, I guess that the full picture
continues to offer us some warnings from history.
At one level the issue "Given that many of the subsequent schema
enhancements then werent implemented or simply couldnt be becuase they
didnt fit the Java brokers model, be it due to the virtualhost
disparity, protocol differences, or lower level things like the memory
management " essentially still exists (albeit without a direct schema
coupling).
One of the things that worries me a little is that in many ways (given
my far from perfect understanding) QMF2 and AMQP 1.0 Management Spec.
share many similarities (no mandatory schema - it's all introspectable
being encoded as AMQP Maps) that's great and offers the ability to align
well at the protocol level, but the differences in the management models
between the two Qpid brokers and of course with other AMQP brokers and
services will continue to make developing cohesive tooling a challenge.
That's life I guess, I guess that you can't expect
Microsoft/IBM/RabbitQ/ActiveMQ/Qpid/etc to offer identical feature sets
but AMQP 1.0 management isn't going to fix that. As you say the Java
Broker has evolved at lot and you've got a lot of quite low level
features in the Management Model that are exposed in the embedded Web
GUI. What I'm not clear on is how far your thinking has progressed along
the options for eventually "externalising" this via AMQP 1.0 Management
and making it available on "external" tooling. That's *largely* what
I've done in the QMF2 space - albeit incomplete/imperfect/whatever, as I
say I think that there are a lot of similarities between QMF2 and AMQP
1.0 Management and a big part of my motivation was proving that it was
possible and exposing the sorts of wrinkles that we might expect (in
particular I guess that I've only exposed the lowest common denominator).
Perhaps the generality of AMQP 1.0 and the needs of Vendors to make
their product sets readily differentiable might mean that we'll never
hit the Nirvana of one Vendor's tooling managing another Vendor's broker
- I mentioned to Rob back at Easter that I worried that the lack of
schemas might make this a lot harder too. I wouldn't want to be coupled
to a schema like QMF1 is, but having a schema available might make it
easier to create generic tooling that is *semantically* meaningful.
Without it the most we can expect from a generic tool is to be a bit
like a JMX Console, which is OK but a bit ugly.
Thanks again for the background, it's really interesting and I think
knowledge of the history is incredibly useful to help us (try to) avoid
the same wrinkles going forward.
Frase
On 24/01/14 01:11, Robbie Gemmell wrote:
> On 23 January 2014 23:54, Fraser Adams <fr...@blueyonder.co.uk>wrote:
>
>> Hey Davin,
>> Robbie has hit the nail on the head. One somewhat unfortunate thing that
>> you probably ought to be aware of is that the management and monitoring for
>> the Qpid C++ and Java Brokers got somewhat erm "fragmented" for various
>> reasons back in the mists of time.
>>
>>
> Just to clarify things a little since much of this was before your (and to
> a smaller extent, even my own) time with the project, it was probably less
> that they became fragmented and more that they started that way. The Java
> broker initially had JMX based management functionality (as is/was popular
> for Java things to do) satisfying its model, and as the C++ broker
> developed it obviously needed managmement tooling to go with it, but didnt
> have the luxury of JMX built right into its runtime platform...though did
> have a nice AMQP interface sitting there, and so QMF grew to fill that need
> and beyond. Thus on the managemnet side things essentially started out
> different and then grew from there.
>
> In early 2010 we did add some initial support for QMF1 to the Java broker
> with aim of moving them closer together, but while we were still working on
> building that out the development on QMF itself moved away from an update
> of its existing binary protocol and instead toward creating what became the
> largely-new QMF2, making for a semi-start-over scenario. We didn't get
> round to that and the incomplete QMF1 support also withered over time,
> causing pain here and there due to continued enhancement of the C++ broker
> and the management schema then leading to breakages in the Java build.
> Given that many of the subsequent schema enhancements then werent
> implemented or simply couldnt be becuase they didnt fit the Java brokers
> model, be it due to the virtualhost disparity, protocol differences, or
> lower level things like the memory management, and in the end we removed
> the QMF1 support (or more accurately didnt bringing it along with us when
> we would have had to completely overhaul it to do so) at the time we begun
> to modularize the brokers management support out to plugins and started
> working on overhauling the configuration model toward its current form.
> Part of that change was us turning thoughts towards an embedded web
> management GUI to facilitate removing the Eclipse RCP based management
> console that existed previously, which then lead us to adding a REST API to
> power it (as has become a more popular way of managing things) which simply
> got based directly around the brokers new configuration model...and that
> about sums up how things ended up as they are now. As you later mention, we
> will be looking to add AMQP Management based functionality in time as that
> specification progresses.
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: QMF REST API - was Re: How to use lifetime policies
Posted by Robbie Gemmell <ro...@gmail.com>.
On 23 January 2014 23:54, Fraser Adams <fr...@blueyonder.co.uk>wrote:
> Hey Davin,
> Robbie has hit the nail on the head. One somewhat unfortunate thing that
> you probably ought to be aware of is that the management and monitoring for
> the Qpid C++ and Java Brokers got somewhat erm "fragmented" for various
> reasons back in the mists of time.
>
>
Just to clarify things a little since much of this was before your (and to
a smaller extent, even my own) time with the project, it was probably less
that they became fragmented and more that they started that way. The Java
broker initially had JMX based management functionality (as is/was popular
for Java things to do) satisfying its model, and as the C++ broker
developed it obviously needed managmement tooling to go with it, but didnt
have the luxury of JMX built right into its runtime platform...though did
have a nice AMQP interface sitting there, and so QMF grew to fill that need
and beyond. Thus on the managemnet side things essentially started out
different and then grew from there.
In early 2010 we did add some initial support for QMF1 to the Java broker
with aim of moving them closer together, but while we were still working on
building that out the development on QMF itself moved away from an update
of its existing binary protocol and instead toward creating what became the
largely-new QMF2, making for a semi-start-over scenario. We didn't get
round to that and the incomplete QMF1 support also withered over time,
causing pain here and there due to continued enhancement of the C++ broker
and the management schema then leading to breakages in the Java build.
Given that many of the subsequent schema enhancements then werent
implemented or simply couldnt be becuase they didnt fit the Java brokers
model, be it due to the virtualhost disparity, protocol differences, or
lower level things like the memory management, and in the end we removed
the QMF1 support (or more accurately didnt bringing it along with us when
we would have had to completely overhaul it to do so) at the time we begun
to modularize the brokers management support out to plugins and started
working on overhauling the configuration model toward its current form.
Part of that change was us turning thoughts towards an embedded web
management GUI to facilitate removing the Eclipse RCP based management
console that existed previously, which then lead us to adding a REST API to
power it (as has become a more popular way of managing things) which simply
got based directly around the brokers new configuration model...and that
about sums up how things ended up as they are now. As you later mention, we
will be looking to add AMQP Management based functionality in time as that
specification progresses.
> The C++ Broker uses Qpid Management Framework (QMF), but that has had an
> interesting evolution and at some point in history QMFv1 migrated to QMFv2
> which caused issues for the Java Broker and they ended up going down a
> different path.
>
> I primarily use the C++ broker, but conversely my team mainly use Java, so
> I started out doing a Java implementation of the QMF2 API without knowing
> much of that background. There was a bit of comedy 'cause the Qpid Java
> guys didn't take a lot of notice of what I was doing 'cause it was QMF and
> the C++ guys didn't really notice 'cause it was Java :-D I was a bit in no
> mans land really :-/
>
> It wasn't really until I put the GUI together that anyone really noticed
> and the penny dropped.
>
> So as Robbie says the REST API that I put together is *actually* a REST
> binding for QMF2 and (apart from being a REST API) doesn't bear too much
> relationship to the Java Broker's REST interface (for most of history we
> were largely unaware of what each other was up to).
>
> It's a far from ideal position and Rob, Gordon, Robbie and I have chatted
> a fair bit about the need to get a more cohesive view. Ultimately the right
> answer will be to adopt the AMQP Management Specification across the board,
> though that's likely to be some way off (I certainly need to find some time
> to properly get my head around it).
>
> By way of "making a stand for the unity of Qpid Brokers :-)" I put
> together a QMF2 plugin for the Java Broker, it was one of those things that
> one does "on principle" - really it was as much as anything a statement to
> say "really guys it *is* actually possible to provide a cohesive view". One
> thing to bear in mind with the Java Broker QMF plugin is that I've tried
> fairly hard to map as much as possible from the Java Broker internal
> Management Model to QMF Management Objects, but there are differences in
> the Management Models, so there are certainly more itty bitty things that
> can be controlled by the Java Broker's native management interface than via
> QMF (mainly 'cause Robbie keeps adding features ;->) but OTOH pretty much
> anything you can do with qpid-config will work with both the C++ and Java
> Brokers.
>
> So to get back to the QMF REST API, the best place to look is the JavaDoc
> (yes really, there is actually a ton of JavaDoc for this stuff)
> start in <qpid>/tools/src/java/docs/api/index.html
> Then look for package org.apache.qpid.restapi and class QpidServer
>
> To get you started if you do (assuming the same host and port as below):
> http://127.0.0.1:8080/qpid/connection/default/console/objects/queue
>
> You will retrieve a list of all of the QMF Queue Management Objects in the
> form of a JSON list (you can do it in a browser too)
> http://127.0.0.1:8080/qpid/connection/default/console/objects/exchangewill retrieve the exchanges
> http://127.0.0.1:8080/qpid/connection/default/console/objects/connectionwill retrieve the connections
> and so on.
>
> the <default> bit is a "convenience", so this is a "true" REST API that
> mirrors the QMF API pretty closely so you really ought to first PUT a QMF
> Console Connection before you do anything else, but of course PUT is a pain
> from a browser, so if you specify "default" for the connection part of the
> URI the API transparently creates (or uses an already created) default QMF
> Console instance.
>
> The GUI actually has a JavaScript class that provides a JavaScript
> implementation of the QMF2 API, which is backed by the REST API, so the GUI
> is generally abstracted from the REST API, there's a factory method for
> that which does the HTTP PUT "under the hood"
>
> You can also use POST in order to invoke QMF methods (for adding/deleting
> queues etc.).
> To be honest I'm a bit rusty - I tend to use the GUI or at worse the
> JavaScript wrapper which abstracts the detail,
> and I've not done it "the hard way" for the best part of 15 months, so the
> following is pulled out from the code.
>
> The POST syntax is:
>
> POST: <host>:<port>/qpid/connection/<name>/object/<ObjectId>
> HTTP body: {"_method_name":<method>,"_arguments":<inArgs>}
> <method>: A string containing the QMF2 method name e.g.
> "getLogLevel", "setLogLevel", "create", "delete".
> <inArgs>: A JSON string containing the method arguments e.g.
> {"level":"debug+:Broker"} for setLogLevel.
> HTTP response: A JSON string containing the response e.g.
> {"level":"notice+"} for getLogLevel (may be empty).
>
> This method invokes the QMF2 method <method> with arguments <inArgs>
> on the object <ObjectId>
>
> And it works like QMF, so what you'd first need to do is to recover the
> Broker Management Object
> http://127.0.0.1:8080/qpid/connection/default/console/objects/broker
>
> You need to do this because the majority of the QMF CRUD methods are
> applied on the Broker Management Object,
> having done this you'd need to do a POST as above (using the ObjectId from
> your own Broker Object)
> POST http://127.0.0.1:8080/qpid/connection/default/object/@0@
> org.apache.qpid.broker:broker:amqp-broker
> And in the HTTP body you need a JSON string.
>
> To Add a queue I *think* that the JSON body goes something like this (for
> your davin_test2 queue):
>
> {
> "_method_name":"create",
> "_arguments": {"type": "queue", "name": "davin_test2", "properties":
> {"durable": true}}
> }
>
> Clearly this isn't something that one would *generally* do "by hand".
>
> I've just done:
>
> curl -X POST -u admin:admin -d '{"_method_name":"create","_arguments":
> {"type": "queue", "name": "davin_test2", "properties": {"durable": true}}}'
> http://127.0.0.1:8080/qpid/connection/default/object/@0@
> org.apache.qpid.broker:broker:amqp-broker
>
> On my system and it created a queue called "davin_test2", so I think what
> I've just said is accurate, yay!!
>
>
> The available properties for Queues and exchanges isn't brilliantly
> documented in the QMF Management Schema
> (or anywhere really) and I kind of "reverse engineered" them, looking
> through the GUI code (qmf-ui.js in qmfui.AddQueue method)
> I can see:
> qpid.max_size
> qpid.max_count
> qpid.flow_stop_size
> qpid.flow_stop_count
> qpid.flow_resume_size
> qpid.flow_resume_count
> durable
> qpid.file_size
> qpid.file_count
> qpid.policy_type
> qpid.last_value_queue
> qpid.last_value_queue_no_browse
> qpid.queue_event_generation
> alternate-exchange
>
> If you look in the qpid-config code I think you'd come across these too.
>
> I've *probably* just made your head explode, so I'd better stop there,
> *hopefully* this is helpful and useful background.
>
> Best regards,
> Frase
>
> P.S. Glad that you like the QMF GUI - it looks pretty nice on an iOS
> device, it uses CSS3 animations so the transitions are GPU accelerated.
>
>
> On 23/01/14 19:42, Robbie Gemmell wrote:
>
>> On 23 January 2014 19:27, Shearer, Davin <ds...@novetta.com> wrote:
>>
>>
>>> [davin@laptop24 qpid]$ curl -X GET -u admin:admin
>>> http://127.0.0.1:8080/rest/queue/davin_test
>>> 404 Not Found: File /rest/queue/davin_test not found.[davin@laptop24qpid
>>> ]$
>>>
>>> Darn, that didn't work... can we put new queues?
>>>
>>> [davin@laptop24 qpid]$ curl -X PUT -u admin:admin -d '{"durable":true}'
>>> http://127.0.0.1:8080/rest/queue/default/davin_test2
>>> 405 Bad Method.[davin@laptop24 qpid]$
>>>
>>> What am I missing?
>>>
>>> Those URLs look a lot like ones that would be used against the REST API
>> from the Java brokers management-http plugin. The URLs/payloads used by
>> the
>> 'QMF REST API' from the tools Fraser put together aren't likely to be that
>> similar to them.
>>
>> I'll now have to defer to someone who can tell you what they would be
>> like...
>>
>> Robbie
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>