You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Gordon Sim <gs...@redhat.com> on 2013/04/01 11:03:05 UTC
Re: Questions from a novice
On 03/29/2013 04:35 PM, Bill Freeman wrote:
> I'm actually using RH MRG, but that seems to be little different from
> upstream Qpid. I'm certainly seeing v2 qmf queues mentioned in the broker
> __dict__. Please say if I'm making unwarranted assumptions here. I wanted
> my code to be as generic as possible, so I'm trying to avoid depending on
> any RH specializations.
There should be no change to the QMF formats or protocols. The only
difference as compared with whatever Qpid release on which it is based
would be possibly a bug fix or two, but those will in any case be in the
Qpid codebase also now.
[...]
> That's good to know. I have been using str(record.objectId()), where
> record is the second argument (third if you count self) to an objectProps()
> method on a sub-class of qmf.console.Console. I'm guessing that the final
> number in this hyphen separated string of digits, where the last number
> tends to be the only multi-digit one, and comes from the objectName field
> of qmf.console.ObjectId instances, is a hash of the information that you
> mention?
> Or am I confusing two separate meanings of "object ID"?
> If so, how to I get the ID or which you speak.
> If not, doesn't hashing to such a small number have collision potential?
That console API is a bit of a mixture of QMFv1 and QMFv2. I think there
is a getObjectId() method, and you can then call asMap() on that to get
the v2 form.
Ken, Ted, can either of you correct or confirm on this? (i.e. how to get
the v2 style package:class:id identifier for a given QMF object using
the old console API.
[...]
> I just didn't fine descriptions of what these were in the docs, though I'm
> calling functions that require them. If there's somewhere I should have
> looked, I'd appreciate a pointer.
Ah, sorry, the package is org.apache.qpid.broker, the class names are
indeed Exchange and Queue.
> I think that you're right on track. I should probably give a bit of
> background on my project as a context for understanding my questions. Or
> ignore the rest of this if not interested:
>
> I'm producing a monitoring tool to help spot issues in a federation having
> many queues. This may sound similar to Cumin, but we want displays in
> different forms (bar graphs of queue message count compared to some defined
> "full" constraint, color changes to attract the administrator's eye to
> queues that are "in trouble", popups to allow browsing messages, looking at
> bindings, move messages from queue to queue, group queues on the page in
> configurable ways... I looked at modifying cumin, but:
> 1. It seems a bit complex - I think that I can write my own faster;
> 2. we don't want to use PostgreSQL (I like pg, but it's not on the
> approved tools list);
> 3. Cumin has a lot of features we don't need; and
> 4. I've caught Cumin missing updates, i.e.;, I spout or drain, sometimes
> the queue info shows up within 10 seconds (update/heartbeat rate), but
> sometimes it takes another change or two before it shows up.
>
> I have a strawman running just showing queue fullness with a few percent of
> full color changes, and now I'm trying to move on to cover the federation.
> I wound up with a Cumin like architecture: A python based qmf console
> process that publishes updates to a redis pubsub channel, subscribed to by
> a tornado based web server, which pushes them on out to browser based
> clients using WebSockets. There's a custom jQueryUI widget per queue in
> the browser that handles display and provides for a context menu.
> WebSockets is inherently bi-directional, so I have another redis pubsub
> channel for things like commands to move messages, requests to grab a
> browse of a queue, get bindings, etc. I'm using tornado because it
> supports WebSockets out of the box, and there's a redis extension for it
> using its event loop, so the web server manages to eschew threads. The
> architecture allows for load balancing by having multiple web servers
> and/or multiple qmf console processes (though I intend that one qmf console
> process be able to handle multiple browser connections, since the
> underlying code seems to support that).
Very interesting. Btw, just in case you haven't seen it, Fraser Adams
has developed a graphical QMF based console as well which is soon to be
added to the repo, but is currently available from
https://issues.apache.org/jira/browse/QPID-3675.
It doesn't (yet!) provide the features you mention above, which do sound
very nice, but perhaps worth comparing notes at some point.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 05/04/13 19:27, Ken Giusti wrote:
> Your code is fine, and your observations are 100% correct. You
> _should_ be seeing both, but are just getting V1 updates. That's the
> bug from QPID-4689. I _just_ pushed a fix to trunk, so if you update,
> you should start seeing V2 objects. The commit is here:
> http://svn.apache.org/viewvc?view=revision&revision=1465050
Thanks Ken - I was questioning my sanity :-) TBH I noticed from earlier
in this thread that you had a patch in play but there's been a lot going
on in this thread and I had assumed that the patch was related to the
issue where if you did "--mgmt-qmf1 no" you didn't get any updates I was
using the broker defaults so I kind of "parked it" not realising that
there was more to it :-)
>>
>> Yes, getting both is awkward at best. The root of the problem is that, by default, brokers send two updates for each event - one in V1 format and one in V2. The reason is to allow for backward and forward compatibility among clients built for v1 and v2 serviced by the same broker.
Don't the V1 and V2 updates get published to different topics? I thought
that qmf.default.topic was for V2 and qpid.management was for V1 so
*surely" it ought to be possible to select.
Thinking about it given the bug QPID-4689 seems to suggest that V2
updates have never actually been delivered hitherto I'm thinking to
myself that perhaps defaulting to V1, but with a new flag/switch
whatever to select V2 in the API is the best approach.
I'd strongly suspect that applying your QPID-4689 fix and thus causing
callbacks from both V1 and V2 is *very* likely to break existing code. I
know that an "isV2" property on the ObjectIds has been mentioned, but
how likely do you think it is that clients are likely to test for this,
especially given that V2 updates are not likely to have actually occurred.
>>
>> I think that we should do two things to address this:
>>
>> 1) start deprecating V1 in the broker by simply changing the default behavior to only send updates as qmf V2. V1 format updates can be turned on manually if needed.
>>
>> 2) fix the console by adding a configuration option to the Console to ignore V1 format object updates. The default would have to allow both, as some clients may actually want both if some of their agents only speak v1.
>>
>> I'd like to propose we do #1 after the next release (for 0.24). What do you think?
>>
Hmm much as I might like to deprecate V1 (and I know I only ever really
use V2 in anger from Java) but my guess is that there's a fair number of
python based QMF tools out there and given that it looks likely that V2
updates have never arrived prior to your fix I really think that your
proposed approach is in danger of breaking an unknown but non-trivial
number of tools. I know that the tool I've been messing with to try this
is a connection-audit tool which checks subscriptions queues against a
whitelist and alerts if someone tries to create a queue not in the
whitelist - it's a bit dear to my heart that one :-)
As I said earlier my preference would be for the API to default to V1
and receive only V1 updates and thus essentially behave as it does at
present prior to QPID-4689.
I think adding a switch to select V2 (I guess to the Session constructor
assuming that varargs are supported so not specifying doesn't break
existing clients) would be the right way IMHO.
I'm kind of happy with "start deprecating V1 in the broker by simply
changing the default behavior to only send updates as qmf V2" but given
that V2 updates to this API appear never actually to have hitherto
happened (if my understanding of QPID-4689 is correct) I think that some
more notice is needed - as I say I think that as proposed existing
clients are likely to break in I suspect subtle ways.
RE "fix the console by adding a configuration option to the Console to
ignore V1 format object updates. The default would have to allow both,
as some clients may actually want both if some of their agents only
speak v1. " As I say I think that the default really should be V1 only
because that's been the situation up until your fix.
One possibility may be to detect the V2 exchanges as is currently done
and automatically only select V2 updates but map them to the
objectProps/objectStats - as said previously at the moment V1 and V2
updates behave differently so making things only do objectProps as seems
to be the case for V2 will break things just about as much as sending
both V1 and V2 updates.
Personally I think it's cleaner to go with my earlier suggestion of
explicitly selecting V2. Frankly when it comes to publicising the
changes that might be necessary when finally deprecating V1 it might be
a whole lot easier to explain to people by pointing to some specific API
changes that they might need to invoke. I have to say that I've never
liked that API at the best of times and the slightly awkward mish-mash
of V1 and V2 I think is an accident waiting to happen.
It's going to be even more exciting when we start to migrate from QMF to
AMQP 1.0 management when that arrives. I think ultimately that's going
to be a good thing, but the transition path is going to be exciting :-)
especially if - like me - you'll have to cater for managing a "mixed
economy" of older and newer brokers.
Cheers,
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Ken Giusti <kg...@redhat.com>.
Hi Frase
----- Original Message -----
> From: "Fraser Adams" <fr...@blueyonder.co.uk>
> To: users@qpid.apache.org
> Sent: Friday, April 5, 2013 2:09:12 PM
> Subject: Re: Questions from a novice
>
> Hi Guys,
> This thread has intrigued me a bit.
>
> I must admit that I had thought the qmf.console.Console,
> qmf.console.Session stuff was QMF1 only so I was intrigued by the
> comments about QMF2. I dug out an old python program.
>
> It starts like:
>
> class AuditConsole(Console):
> # AuditConsole Constructor
> def __init__(self, whitelistFile):
> self.whitelistFile = whitelistFile
> self.whitelistLastModified = None
> # Declare dictionary to map queue name to queue objects
> self.queueNameMap = {}
> # Declare dictionary to map queue name to subscription objects
>
> ......
>
> so it's subclassing Console and using the asynchronous approach. I
> initialise it with:
>
> console = AuditConsole(options.whitelist)
> session = Session(console, rcvObjects=True, rcvEvents=True,
> userBindings=True, manageConnections=True)
>
>
> session.bindClass("org.apache.qpid.broker", "queue")
> session.bindClass("org.apache.qpid.broker", "exchange")
> session.bindClass("org.apache.qpid.broker", "binding")
> session.bindClass("org.apache.qpid.broker", "connection")
> session.bindClass("org.apache.qpid.broker", "session")
> session.bindClass("org.apache.qpid.broker", "subscription")
>
> brokers = []
> try:
> try:
> for host in arguments:
> brokers.append(session.addBroker(host, None,
> options.sasl_mechanism))
>
>
>
> I've got code
>
>
> def objectStats(self, broker, record):
> classKey = record.getClassKey()
> oid = record.getObjectId()
>
> # Note that queue objects are keyed by both queue name and
> Object ID
> if classKey.getClassName() == "queue":
> print "queue: %s" % (oid.asMap())
>
>
> def objectProps(self, broker, record):
> classKey = record.getClassKey()
> oid = record.getObjectId()
>
> # Note that queue objects are keyed by both queue name and
> Object ID
> if classKey.getClassName() == "queue":
> print "queue: %s %s" % (record.name, oid.asMap())
>
>
> but I only seem to be getting QMF1 push updates given the ObjectId
>
>
> queue: qmfc-v2-hb-zappa.4505.1 {'_object_name': '25', '_agent_name':
> '0', '_agent_epoch': 1662}
> queue: {'_object_name': '25', '_agent_name': '0', '_agent_epoch': 1662}
> queue: qmfc-v2-ui-zappa.4505.1 {'_object_name': '23', '_agent_name':
> '0', '_agent_epoch': 1662}
> queue: {'_object_name': '23', '_agent_name': '0', '_agent_epoch': 1662}
> queue: qmfc-v2-zappa.4505.1 {'_object_name': '21', '_agent_name': '0',
> '_agent_epoch': 1662}
> queue: {'_object_name': '21', '_agent_name': '0', '_agent_epoch': 1662}
> queue: reply-zappa.4505.1 {'_object_name': '14', '_agent_name': '0',
> '_agent_epoch': 1662}
> queue: {'_object_name': '14', '_agent_name': '0', '_agent_epoch': 1662}
> queue: topic-zappa.4505.1 {'_object_name': '18', '_agent_name': '0',
> '_agent_epoch': 1662}
> queue: {'_object_name': '18', '_agent_name': '0', '_agent_epoch': 1662}
>
> queue: {'_object_name': '23', '_agent_name': '0', '_agent_epoch': 1662}
> queue: {'_object_name': '18', '_agent_name': '0', '_agent_epoch': 1662}
>
> queue: {'_object_name': '23', '_agent_name': '0', '_agent_epoch': 1662}
> queue: {'_object_name': '18', '_agent_name': '0', '_agent_epoch': 1662}
>
>
> I'm running qpidd 0.23 with no options so both QMF1 and QMF2 data push
> should be available.
>
> What's even odder is that clearly object name 23 relates to
> qmfc-v2-ui-zappa.4505.1 which is, given the name, one of the QMF2 queues.
>
> I'm not at all sure why I should be seeing V1 ObjectIds
>
Your code is fine, and your observations are 100% correct. You _should_ be seeing both, but are just getting V1 updates. That's the bug from QPID-4689. I _just_ pushed a fix to trunk, so if you update, you should start seeing V2 objects. The commit is here:
http://svn.apache.org/viewvc?view=revision&revision=1465050
>
> I'm not particularly familiar with this API (nor really with python).
> The impression I've got from Bill's last post is that he's seeing both
> the QMF1 and QMF2 updates (that seems wrong anyway, I don't think being
> notified for each QMF version is what anyone would expect) but I only
> seem to be seeing QMF1 ObjectIds albeit relating to what appears to be
> QMF2 queues??????
Yes, getting both is awkward at best. The root of the problem is that, by default, brokers send two updates for each event - one in V1 format and one in V2. The reason is to allow for backward and forward compatibility among clients built for v1 and v2 serviced by the same broker.
I think that we should do two things to address this:
1) start deprecating V1 in the broker by simply changing the default behavior to only send updates as qmf V2. V1 format updates can be turned on manually if needed.
2) fix the console by adding a configuration option to the Console to ignore V1 format object updates. The default would have to allow both, as some clients may actually want both if some of their agents only speak v1.
I'd like to propose we do #1 after the next release (for 0.24). What do you think?
>
> Any thoughts??
>
> Frase
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: QMFv2 object update bug [WAS Re: Questions from a novice]
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 11/04/13 16:05, Ken Giusti wrote:
>> One thing though, I think that we discussed on the main thread (and Bill
>> mentioned this too) about providing an extension to specify the QMF
>> version to consume.
> To be frank - I think any effort expended going forward to make it easier to have both QMFv1 and QMFv2 work well together is an effort wasted given QMFv1 deprecation:
>
> http://qpid.2158936.n2.nabble.com/QMF-and-Broker-Management-td7151372.html
>
Yes/no/maybe/possibly :-)
I think that frankly it's an awful lot more complex than that. When Ted
suggested that over a year ago I was very much onside with that view,
I've primarily used QMF2 and I think that there is a lot of fun things
that could potentially be done making interesting QMF2 Agents.
One of the hurdles that I think exists is the lack of cohesion around
QMF. So as I've mentioned before I felt a Java implementation was
missing, so I took it upon myself to do a complete Java implementation
of the draft QMF2 API that was documented (I, not unreasonably, assumed
that this was what the future was going to be because there was nothing
written nor said to the contrary). As it happens there exists a
"prototype" python implementation of that API (which I suspect few, if
any people are actually using) plus an entire miscellany of QMF2 "pseudo
APIs" for python, C++, ruby that generally bear no relationship to each
other (you see where I'm going with this :-))
Moreover from conversations that I've had with various people a lot of
the QMF stuff was perhaps put out without necessarily a lot of community
discussion (that all predates my involvement with the project so maybe
I'm speaking out of turn there), so as a result it has become rather
fragmented - QMF used in the C++ Broker and it's the only game in town
there, the Java Broker has gone its own way.
All this has to be blunt frankly wound me up :-D, and I've tended to get
on my soap box about cohesion and "compelling platforms".
Personally I think QMF2 for all it's failings is pretty good, but the
lack of consultation and cohesion have made it very C++ broker-centric
and thus not a compelling platform across even the Qpid world let alone
the general AMQP world.
So actually I think a decent amount of energy should really be focussed
on the broader AMQP 1.0 Management piece that is currently being thought
about.
However..... given that clearly there is an awful lot of QMF1 and QMF2
tooling out there (and indeed I myself have invested an awful lot of
personal energy into implementing a decent QMF2 API for Java plus web
GUI and all sorts) I'd say there's a danger that the AMQP 1.0 Management
piece will itself fail to become compelling if it doesn't consider the
complexities of migrating the exiting tooling (the QMF1 to QMF2 aspects
in the python API being discussed in this thread are really just a
flavour of the pain to come).
TBH all this is the primary reason that I put together a
QmfManagementPlugin for the Java Broker. It's early days yet and it's
far from perfect, but it was done on a point of principal really that
starting out in the direction of a cohesive management approach is going
to make it a darned sight easier to evolve towards AMQP 1.0 Management.
I really think that we have to play a lot more nicely together as a
community when it comes to management aspects, it's really far too
fragmented. For most large Enterprises it's one of the primary factors
in the decision making process for selecting messaging products. To put
it bluntly Qpid is a great set of products, but we're really doing
ourselves a great disservice if we don't get our act together on
management and monitoring.
So in short I sort of agree that doing too much with QMF1 might be
wasted effort, but not quite for the reasons you suggest - but
conversely (and slightly awkwardly) I suspect most of the current
tooling that exists in the real world (with the exception of my cool
stuff :-)) is I suspect still most likely to be be written using QMF1 -
so it's a conundrum, do we deprecate QMF1 early and risk breaking a lot
of existing Qpid and 3rd party tooling, forcing people to QMF2 when our
strategy should probably be towards AMQP 1.0 Management. or do we retain
QMF1 (possibly "semi"-deprecating it by disabling the push by default in
the broker) but encourage people to use things like qpidtoolib/my
stuff/whatever with the aim of trying to keep that tracking the current
thinking with AMQP 1.0 Management?
TBH I don't know the right answer. I know I quite like the stuff that
I've done and I know that I'm supportive of the stuff Rob is leading on
for AMQP 1.0 Management (even if there are currently bits of it that
give me apoplexy :-)) but trying to build cohesion and build a good
migration roadmap is going to be something of a challenge given the
current level of fragmentation.
I'll step gently off my soap-box now as this has turned into more of a
rant than I intended :-) at the very least it's an interesting topic of
conversation and strategic thinking on Qpid management is long overdue IMHO.
> Agreed - but the sooner we turn off unsolicited QMFv1 updates as planned, the sooner the broker stops sending the same data twice.
True, and the sooner most current Qpid 3rd party management tooling is
likely to break I would suspect :-/
Hopefully my rant above makes some sense, don't get me wrong I don't
much like the idea of prolonging support for QMF1, but as I say I think
it's more complicated than you are suggesting largely because we as a
community haven't done a terribly good job nailing our colours to the
mast and setting an agreed roadmap for management. It's now reached the
point of coming back to haunt us :-/
>
> Yes, you'll see that the broker sends all statistics for a given object rather than just those that changed. While the QMFv2 spec may indicate only the statistics that have changed should be sent - this is very impractical in practice. That spec should be re-written to say "-may- send only those that have changed".
>
> To send only those stats that have changed will require the broker to perform lots of extra work that will end up impacting the performance of its primary purpose - transferring messages. Think about how we'd have to implement this - the broker would have to save a snapshot of every object's statistics between update intervals, then check every single counter for a difference at the next poll (all under a lock, btw). Imagine the impact this would have on the broker's performance, especially as we scale out queues, exchanges, connections, etc. Not worth trading off for mgmt application convenience, IMHO.
>
> -K
>
Ta for that, I knew that there was something weird going on that
affected my (emulated) query subscriptions, I couldn't remember which
way round it was that was only partially being sent. In retrospect it
makes sense that it's the properties, though my previous comment stands
in the general sense. If you look at management-schema.xml for say Queue
there are currently ~35 statistics fields and 7 properties so the saving
of not sending the properties too is probably not proportionate to the
cost of applications having to do a slightly irksome save of the first
update in a Map keyed by ObjectId to get the properties followed by a a
logical "join" of every updated QmfData where we need to look up the
partial state by ObjectId. All just IMHO of course, but it's certainly
one of the reasons I generally do getObjects() style calls - it's a lot
less faff :-)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: QMFv2 object update bug [WAS Re: Questions from a novice]
Posted by Ken Giusti <kg...@redhat.com>.
----- Original Message -----
> From: "Fraser Adams" <fr...@blueyonder.co.uk>
> To: "Ken Giusti" <kg...@redhat.com>
> Cc: dev@qpid.apache.org, "ke1g nh" <ke...@gmail.com>
> Sent: Thursday, April 11, 2013 2:40:36 AM
> Subject: Re: QMFv2 object update bug [WAS Re: Questions from a novice]
>
<snip>
> One thing though, I think that we discussed on the main thread (and Bill
> mentioned this too) about providing an extension to specify the QMF
> version to consume.
To be frank - I think any effort expended going forward to make it easier to have both QMFv1 and QMFv2 work well together is an effort wasted given QMFv1 deprecation:
http://qpid.2158936.n2.nabble.com/QMF-and-Broker-Management-td7151372.html
> For most users it might not be crippling to get the same data twice, but
> it's unwise to presuppose application use cases, I was messing around
> once with a test case covering a fairly large number of consumers (1000
> connections, the same for queues and a few thousand bindings) I hadn't
> really thought much of it previously until I noticed that my MIS traffic
> was taking way more bandwidth than my real traffic (I was monitoring
> queue, exchange and connection traffic) given that the QMF2 data are
> presented as Maps IIRC with a 10 second data interval I was getting MB/s
> :-D (ever come across the Heisenberg Uncertainty Principal :-) I
> definitely assert that it applies as much to data management as quantum
> physics ;->)
>
Agreed - but the sooner we turn off unsolicited QMFv1 updates as planned, the sooner the broker stops sending the same data twice.
>
> Actually that reminds me, you said "V2 handles these updates in a
> different way: it is optimized to generate only one update that combines
> both statistics and properties " - are you sure?
>
Yes - once QMFv2 updates are fixed (as they are on trunk). For example, here are two consecutive updates sent by the broker for a queue named "fleabag":
Object [0-1-1-0-org.apache.qpid.broker:queue:fleabag]
ObjProps PROPS=[[(vhostRef, 0-0-1-0-org.apache.qpid.broker:vhost:org.apache.qpid.broker:broker:amqp-broker,/), (name, 'fleabag'), (durable, False), (autoDelete, True), (exclusive, False), (arguments, {})]]
ObjProps STATS=[[(msgTotalEnqueues, 4), (msgTotalDequeues, 0), (msgTxnEnqueues, 0), (msgTxnDequeues, 0), (msgPersistEnqueues, 0), (msgPersistDequeues, 0), (msgDepth, 4), (byteDepth, 8), (byteTotalEnqueues, 8), (byteTotalDequeues, 0), (byteTxnEnqueues, 0), (byteTxnDequeues, 0), (bytePersistEnqueues, 0), (bytePersistDequeues, 0), (msgFtdEnqueues, 0), (msgFtdDequeues, 0), (byteFtdEnqueues, 0), (byteFtdDequeues, 0), (msgFtdDepth, 0), (byteFtdDepth, 0), (releases, 0), (acquires, 0), (discardsTtl, 0), (discardsRing, 0), (discardsLvq, 0), (discardsOverflow, 0), (discardsSubscriber, 0), (discardsPurge, 0), (reroutes, 0), (consumerCount, 0), (consumerCountHigh, 0), (consumerCountLow, 0), (bindingCount, 1), (bindingCountHigh, 1), (bindingCountLow, 0), (unackedMessages, 0), (unackedMessagesHigh, 0), (unackedMessagesLow, 0), (messageLatencyMin, 0), (messageLatencyMax, 0), (flowStopped, False), (flowStoppedCount, 0)]]
----
Object [0-1-1-0-org.apache.qpid.broker:queue:fleabag]
ObjProps PROPS=[[]]
ObjProps STATS=[[(msgTotalEnqueues, 6), (msgTotalDequeues, 0), (msgTxnEnqueues, 0), (msgTxnDequeues, 0), (msgPersistEnqueues, 0), (msgPersistDequeues, 0), (msgDepth, 6), (byteDepth, 12), (byteTotalEnqueues, 12), (byteTotalDequeues, 0), (byteTxnEnqueues, 0), (byteTxnDequeues, 0), (bytePersistEnqueues, 0), (bytePersistDequeues, 0), (msgFtdEnqueues, 0), (msgFtdDequeues, 0), (byteFtdEnqueues, 0), (byteFtdDequeues, 0), (msgFtdDepth, 0), (byteFtdDepth, 0), (releases, 0), (acquires, 0), (discardsTtl, 0), (discardsRing, 0), (discardsLvq, 0), (discardsOverflow, 0), (discardsSubscriber, 0), (discardsPurge, 0), (reroutes, 0), (consumerCount, 0), (consumerCountHigh, 0), (consumerCountLow, 0), (bindingCount, 1), (bindingCountHigh, 1), (bindingCountLow, 1), (unackedMessages, 0), (unackedMessagesHigh, 0), (unackedMessagesLow, 0), (messageLatencyMin, 0), (messageLatencyMax, 0), (flowStopped, False), (flowStoppedCount, 0)]]
You'll notice the -second- update does not contain any properties, because the broker knows that the properties have not changed since the previous update was sent.
>
> Anyway... and as I say it has been a little while. One thing that I seem
> to recall was that if my query subscription was based on properties all
> was well, but if it included statistics it wasn't doing what I was
> expecting. From memory what I *think* happens is that "one update
> combines both statistics and properties" indeed, but that update only
> contains any statistics that have actually changed.
<snip>
Yes, you'll see that the broker sends all statistics for a given object rather than just those that changed. While the QMFv2 spec may indicate only the statistics that have changed should be sent - this is very impractical in practice. That spec should be re-written to say "-may- send only those that have changed".
To send only those stats that have changed will require the broker to perform lots of extra work that will end up impacting the performance of its primary purpose - transferring messages. Think about how we'd have to implement this - the broker would have to save a snapshot of every object's statistics between update intervals, then check every single counter for a difference at the next poll (all under a lock, btw). Imagine the impact this would have on the broker's performance, especially as we scale out queues, exchanges, connections, etc. Not worth trading off for mgmt application convenience, IMHO.
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: QMFv2 object update bug [WAS Re: Questions from a novice]
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 11/04/13 13:50, Bill Freeman wrote:
>
>
>
> I can see three cases in which a console would want to receive both
> QMF1 and QMF2:
> 1. Testing of QMF itself. But there isn't much that couldn't be
> tested using two separate console processes (arrival order, maybe?),
> so this doesn't seem to be an important case;
possibly, but given that until Ken's update no Console would have
received V2, I'm not convinced.
> 2. If each version offered information that the other didn't. I
> would consider this to be a case of QMF2 being incomplete (QMF1 is
> legacy and need not grow features added by QMF1), in which case the
> correct fix is probably to add the missing features to QMF2, unless
> the data is something that wants to be deprecated;
I'm pretty sure that in terms of information from the actual management
objects QMF1 and QMF2 return the same information if not in the same
format (different callback approaches as previously discussed)
> 3. There is some reason to connect to multiple brokers directly, and
> some of them are not V2 capable. You could, of course, talk to them
> all with V1, unless you were counting on federation among V2 brokers
> to allow you to reach some that would otherwise require you to have an
> explicitly set up tunnel in order to connect to them.
Is that actually possible? To do that I'd have thought that a Console
would need to have multiple Connections, I thought that the Connection
was created when the Console got constructed using the options passed. I
*think* a more likely use case might be where multiple Agents are
registered with some being QMF1 and some QMF2. So say for example you
configure the broker Management Agent to only send in QMF2, but you have
a bespoke Agent built using QMF1. If you are only monitoring the broker
Management Agent though that won't hold.
>
>
> The nice thing about the separate callback for V2 updates is that no
> existing code will break because of receiving unexpected duplicate
> (sort of) updates, since existing code still won't see the V2 updates.
>
> But the network traffic point is well taken. I do prefer filtering to
> be closer to the source.
I wasn't trying to suggest that having a separate callback wasn't a good
idea, I think it's the only really good idea, I was simply suggesting
that in itself it was sub-optimal and that in addition it might be an
idea to consider additional API calls for explicitly selecting QMF2 or
both so that the appropriate subscriptions could be created to minimise
data duplication.
>
>
>
> I suspect that subsequent updates contain any properties WHICH HAVE
> CHANGED.
Not sure what you mean here, it might just be language, the "properties"
as being described here are the largely static things and the statistics
are the things that tend to change (if you look in management-schema.xml
it illustrates what fields are what).
So subsequent QMF2 updates I believe contain all of the properties but
only the specific statistics that have changed since the last update.
I'm fairly sure about that because I'm able to do a query subscription
on things like queue name.
> But since properties don't tend to change, updates after the initial
> *everything* update don't carry any property *updates*. This would
> make it hard to filter based on properties. One doesn't want to carry
> out such filtering at the agent, because then everyone is stuck with
> the same filtering.
I was talking about what is currently actually happening not what
*could* be made to happen, personally I'd simply push the entire object,
it makes application code a whole lot simpler. Sure there's some
bandwidth overhead but from what I've seen the instances where you tend
to see the most objects (Connections and Queues) if one statistic has
changed actually most of them have tended to so you don't actually save
as much as you might think by trying to optimise to only send the
changed statistic (e.g. sending a statistics "diff" plus properties is
what currently happens).
>
> A possible approach is that, in addition to carrying changed
> properties and statistics, QMF2 messages would carry a decoration
> containing all of the properties, available for use by the
> subscription controls, but which is stripped by the time the message
> hits the wire. But that really moves away from the QMF2 messages
> aren't special concept. It's hard to see how the overhead of checking
> for the decoration for stripping could be avoided for bread and butter
> non QMF messages.
As I say what currently happens is that in QMF2 all the (static)
properties are sent each time, but only the updated statistics get sent
- it's been a while since I checked for sure, but that certainly rings a
bell 'cause it seemed a bit weird at the time. And as I also said I
remain to be convinced that "optimisations" in terms of trying to be
clever with the statistics being sent are massively worthwhile.
>
> Another possibility is to limit subscription filtering to being based
> upon that which is carried. Apparently there is enough that the
> console can figure out the object ID (perhaps it is carried itself),
> from which you can parse out the object name at need (or is this only
> true for queues?). Hopefully, if QMF messages can be delivered across
> a federation, there is enough to identify the object's broker.
So my Java QMF2 implementation can do much more sophisticated filtering
than that, it can do boolean combinations of properties, regexes and all
sorts. In theory say I could write a query subscription to only pass say
a specific named queue whose msgDepth had exceeded a particular value
(that's only an example and I'd probably use an event for that but the
point is you can do neat things but the Management Agent only passing
back changed statistics limits some of the filtering choices)
Oh for sure I could probably update my Java QMF2 query subscription code
to retain more state before the filter gets applied, but I had assumed
that the documented QMF2 API would gain traction and that query
subscriptions would be properly implemented in the Management Agent
(doing it client side in the Console implementation is purely a
work-around) so I put it on the backburner as I've already implemented
way more of that API in the Java version than anyone else has and as I
tend to use getObjects() mostly I doubt any one else is using the really
clever bits in anger. It's a shame really as the query subscription idea
in QMF2 is quite neat and powerful.
Given some of the discussions going on on the subject of AMQP 1.0
Management I'm more inclined to invest my time trying to figure out how
to migrate to that with the minimum of disruption and to try and align
the C++ and Java Brokers a little more.
None of the previous couple of paragraphs are relevant to the current
python stuff in case that wasn't totally clear, unfortunately QMF2 the
protocol and QMF2 the (miscellaneous set of) APIs can lead to a world of
confusion, frankly it's less than ideal :-(
Frase
Re: QMFv2 object update bug [WAS Re: Questions from a novice]
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Thanks Ken,
I definitely think it's a far better proposal to provide an API
extension (objectUpdate) than rather than something which is confusing
at best (and more likely a breaking change that will probably break in
subtle ways).
One thing though, I think that we discussed on the main thread (and Bill
mentioned this too) about providing an extension to specify the QMF
version to consume. As it stands it'll subscribe to
qmf.default.topic/data.ind.* or something like that for QMF1 and a
similar subscription off (I think) qpid.management for QMF2 pushing
essentially the same data twice is somewhat wasteful of bandwidth. There
*may* be applications that need to deal with a mixed economy (I'm not
convinced that would be the case from a single connection/console
instance but maybe that's just me) but in any case as we have discussed
"Its likely that apps written using the console haven't ever received v2
style updates" so I'm tempted to suggest that (despite wanting to move
away from QMF1!!) the optimal non-breaking change would be to default to
subscribing to qmf.management and delivering to objectProps/objectStats
with additional API calls to enable QMF2 only and to enable both (I'm
sceptical about this last use case but if it doesn't harm anything else
I don't mind giving the choice).
For most users it might not be crippling to get the same data twice, but
it's unwise to presuppose application use cases, I was messing around
once with a test case covering a fairly large number of consumers (1000
connections, the same for queues and a few thousand bindings) I hadn't
really thought much of it previously until I noticed that my MIS traffic
was taking way more bandwidth than my real traffic (I was monitoring
queue, exchange and connection traffic) given that the QMF2 data are
presented as Maps IIRC with a 10 second data interval I was getting MB/s
:-D (ever come across the Heisenberg Uncertainty Principal :-) I
definitely assert that it applies as much to data management as quantum
physics ;->)
Actually that reminds me, you said "V2 handles these updates in a
different way: it is optimized to generate only one update that combines
both statistics and properties " - are you sure?
It has been over a year since I looked at this (I mostly do synchronous
getObjects() calls) so my recollection is slightly vague but you may or
may not know this but the QMF2 API spec has the notion of "query
subscriptions" which are quite neat. When I implemented this in the Java
implementation of that API I hit a teeny weeny hurdle - basically the
C++ broker Management Agent doesn't actually support that part of the
QMF2 protocol :-D what I did when a query subscription was specified
against the broker Management Agent was to set up a (AMQP/JMS)
subscription to qmf.default.topic/data.ind.* then apply the query
specified as a filter. It was sub-optimal as the data is still being
sent over the network, but from an application perspective it works
pretty well (my Java QpidQueueStat implementation actually used this
approach kind of as a test case and as an API demo).
Anyway... and as I say it has been a little while. One thing that I seem
to recall was that if my query subscription was based on properties all
was well, but if it included statistics it wasn't doing what I was
expecting. From memory what I *think* happens is that "one update
combines both statistics and properties" indeed, but that update only
contains any statistics that have actually changed. It's not *quite* the
same thing as saying "update that combines both statistics and
properties" users of this API will still have to have code very similar
(but not quite the same) as they'd have needed for V1 in order to "join"
and present the complete representation of the QmdData instance.
I'll grant you it's more bandwidth efficient (though I suspect only
marginally given that most stats on the most frequently changing Objects
will be changing) but it's one heck of a lot more confusing to users
behaving the way it does.
As I'm typing my memory is coming back and I'm *pretty sure* that this
is indeed how QMF2 unsolicited data updates are currently working.
Frase
On 10/04/13 16:41, Ken Giusti wrote:
> Folks,
>
> Recently I submitted a fix on trunk that addressed a problem where QMF v2 object update notifications were being dropped by the python QMF console (qmf.console). The fix merely re-enabled the update notifications.
>
> As pointed out by both Fraser Adams and Bill, this may cause problems with existing clients. As described below (and elsewhere in this thread), now both QMFv1 updates and QMFv2 updates will be delivered via the qmf.console.Console callback interface.
>
> Sadly, the way this is done is confusing at best. For V1 updates, there are separate callbacks for statistics updates (via objectStats()) and property updates (via objectProps). The confusion comes from the fact that V2 handles these updates in a different way: it is optimized to generate only one update that combines both statistics and properties. However that update arrives via the objectProps() callback. For QMFv2, the objectStats() callback is ignored. This forces the application's objectProps to have to deal with different update formats - v1 and v2 - and handle the different use cases (updating statistics in the v2 case, not for v1).
>
> AFAIK - this bug has prevented v2 updates from arriving on that objectProps callback since prior to 0.18. Its likely that apps written using the console haven't ever received v2 style updates.
>
> I propose we fix this in the manner suggested by Bill and Fraser. Specifically:
>
> 1) Add a new interface to the qmf.console.Console object:
> def objectUpdate( self, broker, record )
> This callback will be invoked only for QMFv2 object updates. The parameters are analogous to objectProps and objectStats
>
> 2) The existing objectProps and objectStats callbacks are invoked only for the v1 style updates, as was their behavior prior to my fix. v2 object updates will never arrive on these callbacks.
>
> Any objections?
>
>
> ----- Original Message -----
>> From: "Bill Freeman" <ke...@gmail.com>
>> To: users@qpid.apache.org
>> Sent: Friday, April 5, 2013 4:20:37 PM
>> Subject: Re: Questions from a novice
>>
>> On Fri, Apr 5, 2013 at 3:38 PM, Ken Giusti <kg...@redhat.com> wrote:
>>
>>> ----- Original Message -----
>>>> From: "Bill Freeman" <ke...@gmail.com>
>>> QMF V2 has been available for some time now, and we've cut all the project
>>> tools and libraries over to speaking it. If we start down the road to
>>> deprecating V1, turning off the broker updates is about the least risky
>>> first step I can think of. It would not remove v1 at all - but it would
>>> require manual intervention to restore the old behavior. Having to
>>> explicitly take that action would be a (hopefully gentle) nudge to assess a
>>> deployment's V1 dependency.
>>>
>>>
>>> I think the on line documentation may have something to answer for here.
>> When I started this project, since we are using MRG, I started with
>> RedHat's documentation, but didn't find any advice about coding QMF. Then
>> Google got me to the Apache Qpid pages, link to QMF, and eventually
>> dribbled down to an example that became the base of my current design.
>> That was based on qmf.console. There was no insight there about selecting
>> V1 versus V2. There were also several pages, at least, that seemed to
>> imply that V1 was now, and everything would support it, while V2 was new
>> and for the future, and not universally supported (and that turned out to
>> be correct until your patch).
>>
>> I think that the concept that consoles, for some time yet to come, should
>> endeavor to support both versions, is a good one. They should perhaps log
>> a deprecation warning if they discover a V1 only broker (aren't V2 brokers
>> supposed to translate for V1 only agents?), but they should work anyway.
>>
>> One thing that makes it tough for consoles to do so, at least if they are
>> based on qmf.console, is the fact that the objectProps callback is shared
>> between versions, but has a different meaning. One callback carrying
>> deltas is a good design, but it seems that either it should use a
>> completely separate callback, or it should call both objectProps and
>> objectStats (in that order), omitting objectProps if it has no properties,
>> and omitting objectStats if it has no statistics. A V2 specific callback
>> would be my preference, but only by a little. Since the other stuff has
>> never been called for V2 updates before, this is the time to make that
>> change.
>>
>> And I guess that's also a potential problem with making the fix from your
>> patch: Code that has been working for years using qmf.console has never
>> received a V2 objectProps call before may not work with it (mine certainly
>> didn't). A separate callback for V2 updates fixes this issue.
>>
>> Bill
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: Plans for QMF v1 and v2 (was Re: QMFv2 object update bug [WAS Re:
Questions from a novice])
Posted by Bill Freeman <ke...@gmail.com>.
Just a few thoughts:
Since it appears to be the only asynchronous python QMF client, my own
project needs the qmf.console package to continue to work.
I presume that those tools, which Ken mentions as being based on QMFv2 now,
do not use this package.
Now that (the patched version of) it is actually capable of getting QMFv2
updates, I'm happy to use that (and have been). I am concerned, however,
that some of its internals (e.g.; fetching a schema) might depend on
QMFv1. They certainly worked before the patch that enabled v2, so they
must have been using v1. Thus my question is whether, if QMFv1 suddenly
went away, these internal operations would seamlessly switch to using v2,
or would they just break?
What I'd like to see is for someone who knows what they are doing* to patch
qmf/console.py (probably in that same _TryToConnect() method) so that it
does not establish any v1 connections at all, confirm that they think that
it still works, and pass me a copy to test in my project. If all that goes
well, or if the library used by all the tools grows asynchronous capability
(and I learn to use it), then, and only then, would I be comfortable with
the disappearance of any QMFv1 capabilities.
[* That is, not me. As has been pointed out here recently, it uses the man
behind the curtain sections of the Qpid python library, not the giiant face
on the screen "messaging" interface.]
I don't know whether anyone besides me (and the QMF python tutorial I
started with) uses qmf/console.py . I haven't noticed anyone speaking up
during our discussions, so I may be the only one affected. But there could
well be users not reading the list. While they may not have been promised
compatibility across dot releases, it seems a shame to break their code
unless there are real savings to be had. The v1 code exists. Unless other
changes require it to be maintained, or unless it represents a significant
performance drag, why go to the effort of removing it, at least before AMPQ
1.0?
Bill
On Mon, May 6, 2013 at 8:54 AM, Ken Giusti <kg...@redhat.com> wrote:
> Folks,
>
> My overall concern is that this project doesn't have the bandwidth to
> adequately support two different management implementations. That's really
> the crux of the issue for me.
>
> The current problems we've been dealing with come down to the fact that
> there's inadequate test coverage for dual use QMFv1 and QMFv2. The latest
> problem (QMFv2 updates from the broker being dropped by qmf.console) has
> been broken for a long time. We shipped this broken for several releases.
> Why? Inadequate test coverage.
>
> Most - if not all - of the python tests use QMFv1. Yet qpid-tools are now
> (mostly) based on QMFv2. I'm not even sure to what extent the QMF agent
> libraries are tested - perhaps not at all for the cmake builds. This
> results in partial test coverage of both implementations - neither
> implementation gets fully tested.
>
> In the end, we try to support both, but are not able to guarantee the
> level of quality our users deserve for either.
>
> We should be concentrating our efforts on one implementation, and making
> it the best we can.
>
> -K
>
> ----- Original Message -----
> > From: "Gordon Sim" <gs...@redhat.com>
> > To: users@qpid.apache.org
> > Cc: dev@qpid.apache.org
> > Sent: Monday, May 6, 2013 7:11:18 AM
> > Subject: Plans for QMF v1 and v2 (was Re: QMFv2 object update bug [WAS
> Re: Questions from a novice])
> >
> > On 05/03/2013 04:51 PM, Ken Giusti wrote:
> > > I want to know what the
> > > QPID project's plan is for QMFv1/v2 support in the C++ broker going
> > > forward.
> > >
> > > If the consensus is that a 1.x C++ broker is going to support both
> > > QMFv1 and QMFv2, then simply reverting all the changes I've made to
> > > qmf.console and reverting to the old behavior is probably the least
> > > risky move.
> > >
> > > On the other hand, if the consensus is to deprecate QMFv1 support
> > > prior to releasing a 1.0 C++ broker - or perhaps toss in the towel on
> > > QMFv2 - then the final fix for this needs to take that plan into
> > > account.
> > >
> > > So I'm going to call a discussion among the developers, proposing we
> > > vote on what QMF support a 1.x C++ broker will provide.
> >
> > I'd like to move the discussion to the user list as there are folks
> > there who may not be on dev list but would be impacted or might
> > otherwise have an opinion. I'm assuming everyone on the dev list is also
> > on the user list (if not you really should be :-)
> >
> > By way of a short introduction, Ken's question comes after a long thread
> > focused on some issues around the dual support of v1 and v2 of the QMF
> > protocol in the console.py python library for applications built to
> > manage/monitor the c++ broker.
> >
> > > Once that's decided, we can determine the most appropriate fix.
> > >
> > > As an aside - we've talked a lot about preserving behavior and
> > > functionality across 0.X releases in this thread. While nobody wants
> > > to gratuitously break stuff, this project has reserved the right to
> > > remove and change functionality between 0.x releases in order to fix
> > > bugs and provide new features. Things may be different once we get
> > > to 1.x, but for 0.x, this has been the case.
> >
> > For my part, I am in favour of removing QMFv1 support, if necessary
> > providing some sort of adapter or plugin to deal with any use cases for
> > which this might prove problematic. I think QMFv2 is now firmly
> > established as the current management mechanism for the c++ broker.
> >
> > Ultimately I also think we should be starting to think about an AMQP 1.0
> > based management standard and how we would transition to that. That is
> > however a larger question that shouldn't hold up the immediate decision
> > regarding QMFv1 though it may inform it a little.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
> >
>
> --
> -K
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Plans for QMF v1 and v2 (was Re: QMFv2 object update bug [WAS
Re: Questions from a novice])
Posted by Ken Giusti <kg...@redhat.com>.
Folks,
My overall concern is that this project doesn't have the bandwidth to adequately support two different management implementations. That's really the crux of the issue for me.
The current problems we've been dealing with come down to the fact that there's inadequate test coverage for dual use QMFv1 and QMFv2. The latest problem (QMFv2 updates from the broker being dropped by qmf.console) has been broken for a long time. We shipped this broken for several releases. Why? Inadequate test coverage.
Most - if not all - of the python tests use QMFv1. Yet qpid-tools are now (mostly) based on QMFv2. I'm not even sure to what extent the QMF agent libraries are tested - perhaps not at all for the cmake builds. This results in partial test coverage of both implementations - neither implementation gets fully tested.
In the end, we try to support both, but are not able to guarantee the level of quality our users deserve for either.
We should be concentrating our efforts on one implementation, and making it the best we can.
-K
----- Original Message -----
> From: "Gordon Sim" <gs...@redhat.com>
> To: users@qpid.apache.org
> Cc: dev@qpid.apache.org
> Sent: Monday, May 6, 2013 7:11:18 AM
> Subject: Plans for QMF v1 and v2 (was Re: QMFv2 object update bug [WAS Re: Questions from a novice])
>
> On 05/03/2013 04:51 PM, Ken Giusti wrote:
> > I want to know what the
> > QPID project's plan is for QMFv1/v2 support in the C++ broker going
> > forward.
> >
> > If the consensus is that a 1.x C++ broker is going to support both
> > QMFv1 and QMFv2, then simply reverting all the changes I've made to
> > qmf.console and reverting to the old behavior is probably the least
> > risky move.
> >
> > On the other hand, if the consensus is to deprecate QMFv1 support
> > prior to releasing a 1.0 C++ broker - or perhaps toss in the towel on
> > QMFv2 - then the final fix for this needs to take that plan into
> > account.
> >
> > So I'm going to call a discussion among the developers, proposing we
> > vote on what QMF support a 1.x C++ broker will provide.
>
> I'd like to move the discussion to the user list as there are folks
> there who may not be on dev list but would be impacted or might
> otherwise have an opinion. I'm assuming everyone on the dev list is also
> on the user list (if not you really should be :-)
>
> By way of a short introduction, Ken's question comes after a long thread
> focused on some issues around the dual support of v1 and v2 of the QMF
> protocol in the console.py python library for applications built to
> manage/monitor the c++ broker.
>
> > Once that's decided, we can determine the most appropriate fix.
> >
> > As an aside - we've talked a lot about preserving behavior and
> > functionality across 0.X releases in this thread. While nobody wants
> > to gratuitously break stuff, this project has reserved the right to
> > remove and change functionality between 0.x releases in order to fix
> > bugs and provide new features. Things may be different once we get
> > to 1.x, but for 0.x, this has been the case.
>
> For my part, I am in favour of removing QMFv1 support, if necessary
> providing some sort of adapter or plugin to deal with any use cases for
> which this might prove problematic. I think QMFv2 is now firmly
> established as the current management mechanism for the c++ broker.
>
> Ultimately I also think we should be starting to think about an AMQP 1.0
> based management standard and how we would transition to that. That is
> however a larger question that shouldn't hold up the immediate decision
> regarding QMFv1 though it may inform it a little.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: Plans for QMF v1 and v2 (was Re: QMFv2 object update bug [WAS
Re: Questions from a novice])
Posted by Ken Giusti <kg...@redhat.com>.
Folks,
My overall concern is that this project doesn't have the bandwidth to adequately support two different management implementations. That's really the crux of the issue for me.
The current problems we've been dealing with come down to the fact that there's inadequate test coverage for dual use QMFv1 and QMFv2. The latest problem (QMFv2 updates from the broker being dropped by qmf.console) has been broken for a long time. We shipped this broken for several releases. Why? Inadequate test coverage.
Most - if not all - of the python tests use QMFv1. Yet qpid-tools are now (mostly) based on QMFv2. I'm not even sure to what extent the QMF agent libraries are tested - perhaps not at all for the cmake builds. This results in partial test coverage of both implementations - neither implementation gets fully tested.
In the end, we try to support both, but are not able to guarantee the level of quality our users deserve for either.
We should be concentrating our efforts on one implementation, and making it the best we can.
-K
----- Original Message -----
> From: "Gordon Sim" <gs...@redhat.com>
> To: users@qpid.apache.org
> Cc: dev@qpid.apache.org
> Sent: Monday, May 6, 2013 7:11:18 AM
> Subject: Plans for QMF v1 and v2 (was Re: QMFv2 object update bug [WAS Re: Questions from a novice])
>
> On 05/03/2013 04:51 PM, Ken Giusti wrote:
> > I want to know what the
> > QPID project's plan is for QMFv1/v2 support in the C++ broker going
> > forward.
> >
> > If the consensus is that a 1.x C++ broker is going to support both
> > QMFv1 and QMFv2, then simply reverting all the changes I've made to
> > qmf.console and reverting to the old behavior is probably the least
> > risky move.
> >
> > On the other hand, if the consensus is to deprecate QMFv1 support
> > prior to releasing a 1.0 C++ broker - or perhaps toss in the towel on
> > QMFv2 - then the final fix for this needs to take that plan into
> > account.
> >
> > So I'm going to call a discussion among the developers, proposing we
> > vote on what QMF support a 1.x C++ broker will provide.
>
> I'd like to move the discussion to the user list as there are folks
> there who may not be on dev list but would be impacted or might
> otherwise have an opinion. I'm assuming everyone on the dev list is also
> on the user list (if not you really should be :-)
>
> By way of a short introduction, Ken's question comes after a long thread
> focused on some issues around the dual support of v1 and v2 of the QMF
> protocol in the console.py python library for applications built to
> manage/monitor the c++ broker.
>
> > Once that's decided, we can determine the most appropriate fix.
> >
> > As an aside - we've talked a lot about preserving behavior and
> > functionality across 0.X releases in this thread. While nobody wants
> > to gratuitously break stuff, this project has reserved the right to
> > remove and change functionality between 0.x releases in order to fix
> > bugs and provide new features. Things may be different once we get
> > to 1.x, but for 0.x, this has been the case.
>
> For my part, I am in favour of removing QMFv1 support, if necessary
> providing some sort of adapter or plugin to deal with any use cases for
> which this might prove problematic. I think QMFv2 is now firmly
> established as the current management mechanism for the c++ broker.
>
> Ultimately I also think we should be starting to think about an AMQP 1.0
> based management standard and how we would transition to that. That is
> however a larger question that shouldn't hold up the immediate decision
> regarding QMFv1 though it may inform it a little.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Plans for QMF v1 and v2 (was Re: QMFv2 object update bug [WAS Re:
Questions from a novice])
Posted by Rob Godfrey <ro...@gmail.com>.
So, the initial thinking for AMQP Management has (finally) now been
uploaded so that anyone can see it [1]. As Fraser mentioned, the scope of
the AMQP Management work at OASIS is currently purely about "mechanism" and
not defining specific operations / attributes that are available on a
managed object.
-- Rob
[1] https://lists.oasis-open.org/archives/amqp/201305/msg00005.html
On 7 May 2013 20:18, Fraser Adams <fr...@blueyonder.co.uk> wrote:
> On 06/05/13 12:11, Gordon Sim wrote:
>
>>
>> For my part, I am in favour of removing QMFv1 support, if necessary
>> providing some sort of adapter or plugin to deal with any use cases for
>> which this might prove problematic. I think QMFv2 is now firmly established
>> as the current management mechanism for the c++ broker.
>>
> My take on this is that I'm more than happy to see the back of QMF1 but
> I'd be keen to understand whether or not this results in *breaking changes*.
>
> A couple of things I mentioned on the dev group around this point are
> a) The asynchronous python API - that's where this discussion started the
> problem is that the callbacks that have actually worked to date work for
> QMF1, there was a suggestion of an additional objectUpdate() call for QMF2
> which is OK as it wouldn't break existing clients as long as QMF1 data was
> still pushed, but would if it was disabled. Forcing clients to change from
> statUpdate() and propUpdate() to objectUpdate() (and any changes regression
> testing that this involves) is clearly a breaking change. Parochially I've
> only got one tool that uses the async API but I'd prefer it not to be
> broken (mainly because I wrote it at home for a work use and I'm not
> responsible for supporting it and I'd end up having to explain the
> subtleties we're discussing here to someone with a lot less QMF experience
> than me :'( ). At a push as long as I have decent notice I can rewrite it
> myself, but others may not have that luxury - as Bill has mentioned
> previously he's a contractor so when he finishes up his contract and
> someone else takes over his work they won't have this background and it
> might "silently" stop working properly.
> b) If anyone has been writing Agents using QMF1 then clearly they are
> likely to break. Parochially this doesn't affect me, but I'm not sure quite
> how we can gauge how much it may affect others.
>
> Perhaps rather like the automake to cmake change we start off by giving
> gentle nudges then get a bit more forceful, so disabling the QMF1 broker
> updates as Ken has suggested might be a start (along with some decent
> documentation and a deprecation notice in the qpidd -h text).
>
> I'd personally quite like to see the python async API shim the QMF2 update
> to make it behave like statUpdate()/propUpdate() though I accept this might
> be difficult and would have to bow to people who know more about the
> implementation of this than me.
>
>
>
>> Ultimately I also think we should be starting to think about an AMQP 1.0
>> based management standard and how we would transition to that. That is
>> however a larger question that shouldn't hold up the immediate decision
>> regarding QMFv1 though it may inform it a little.
>>
>> Yeah I totally agree, I hope that it's not too long before Rob is able
> to share the draft AMQP 1.0 Management Spec. It'd be good to have a "lively
> discussion" about this among a wider group. Rob and I had a couple of
> decent chats over Easter but I'm keen to start some proper thinking on how
> we can make this a reality. Unfortunately I suspect that the QMF1/QMF2
> quirks are likely to be minor compared to moving to AMQP 1.0 Management,
> but hopefully not insurmountable, I'm hopeful that it should be possible to
> shim between ManagementAgent and ManagementNode and vice versa which should
> help with migration. As we've discussed a pluggable Management layer for
> the C++ broker would be good, it really helped in the Java broker - I was
> able to get QMF2 support up and working in a couple of weekends with no
> prior knowledge of the Java broker code base.
>
> <rant>
>
> As an aside the AMQP 1.0 Management specification is only covering the
> protocol as I understand it, but I'd be quite keen for us to get our act
> together with respect to APIs. Yes I know it's possible to write QMF2 using
> the protocol directly, but frankly it's not as interoperable as it might be
> (C++ binary strings versus UTF8 strings are a personal "favourite" of mine
> resulting in byte[] or String respectively in Java - ClassCastException
> hell :-)). So I think it's useful to have a decent API, however the four or
> five APIs that we seem to have for QMF is less helpful :-D - can we agree
> on one when we start looking at AMQP 1.0 Management?
>
> </rant> :-)
>
> Cheers,
> Frase
>
>
>
>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.**org<de...@qpid.apache.org>
> For additional commands, e-mail: dev-help@qpid.apache.org
>
>
Re: Plans for QMF v1 and v2 (was Re: QMFv2 object update bug [WAS
Re: Questions from a novice])
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 06/05/13 12:11, Gordon Sim wrote:
>
> For my part, I am in favour of removing QMFv1 support, if necessary
> providing some sort of adapter or plugin to deal with any use cases
> for which this might prove problematic. I think QMFv2 is now firmly
> established as the current management mechanism for the c++ broker.
My take on this is that I'm more than happy to see the back of QMF1 but
I'd be keen to understand whether or not this results in *breaking changes*.
A couple of things I mentioned on the dev group around this point are
a) The asynchronous python API - that's where this discussion started
the problem is that the callbacks that have actually worked to date work
for QMF1, there was a suggestion of an additional objectUpdate() call
for QMF2 which is OK as it wouldn't break existing clients as long as
QMF1 data was still pushed, but would if it was disabled. Forcing
clients to change from statUpdate() and propUpdate() to objectUpdate()
(and any changes regression testing that this involves) is clearly a
breaking change. Parochially I've only got one tool that uses the async
API but I'd prefer it not to be broken (mainly because I wrote it at
home for a work use and I'm not responsible for supporting it and I'd
end up having to explain the subtleties we're discussing here to someone
with a lot less QMF experience than me :'( ). At a push as long as I
have decent notice I can rewrite it myself, but others may not have that
luxury - as Bill has mentioned previously he's a contractor so when he
finishes up his contract and someone else takes over his work they won't
have this background and it might "silently" stop working properly.
b) If anyone has been writing Agents using QMF1 then clearly they are
likely to break. Parochially this doesn't affect me, but I'm not sure
quite how we can gauge how much it may affect others.
Perhaps rather like the automake to cmake change we start off by giving
gentle nudges then get a bit more forceful, so disabling the QMF1 broker
updates as Ken has suggested might be a start (along with some decent
documentation and a deprecation notice in the qpidd -h text).
I'd personally quite like to see the python async API shim the QMF2
update to make it behave like statUpdate()/propUpdate() though I accept
this might be difficult and would have to bow to people who know more
about the implementation of this than me.
>
> Ultimately I also think we should be starting to think about an AMQP
> 1.0 based management standard and how we would transition to that.
> That is however a larger question that shouldn't hold up the immediate
> decision regarding QMFv1 though it may inform it a little.
>
Yeah I totally agree, I hope that it's not too long before Rob is able
to share the draft AMQP 1.0 Management Spec. It'd be good to have a
"lively discussion" about this among a wider group. Rob and I had a
couple of decent chats over Easter but I'm keen to start some proper
thinking on how we can make this a reality. Unfortunately I suspect that
the QMF1/QMF2 quirks are likely to be minor compared to moving to AMQP
1.0 Management, but hopefully not insurmountable, I'm hopeful that it
should be possible to shim between ManagementAgent and ManagementNode
and vice versa which should help with migration. As we've discussed a
pluggable Management layer for the C++ broker would be good, it really
helped in the Java broker - I was able to get QMF2 support up and
working in a couple of weekends with no prior knowledge of the Java
broker code base.
<rant>
As an aside the AMQP 1.0 Management specification is only covering the
protocol as I understand it, but I'd be quite keen for us to get our act
together with respect to APIs. Yes I know it's possible to write QMF2
using the protocol directly, but frankly it's not as interoperable as it
might be (C++ binary strings versus UTF8 strings are a personal
"favourite" of mine resulting in byte[] or String respectively in Java -
ClassCastException hell :-)). So I think it's useful to have a decent
API, however the four or five APIs that we seem to have for QMF is less
helpful :-D - can we agree on one when we start looking at AMQP 1.0
Management?
</rant> :-)
Cheers,
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Plans for QMF v1 and v2 (was Re: QMFv2 object update bug [WAS Re:
Questions from a novice])
Posted by Gordon Sim <gs...@redhat.com>.
On 05/03/2013 04:51 PM, Ken Giusti wrote:
> I want to know what the
> QPID project's plan is for QMFv1/v2 support in the C++ broker going
> forward.
>
> If the consensus is that a 1.x C++ broker is going to support both
> QMFv1 and QMFv2, then simply reverting all the changes I've made to
> qmf.console and reverting to the old behavior is probably the least
> risky move.
>
> On the other hand, if the consensus is to deprecate QMFv1 support
> prior to releasing a 1.0 C++ broker - or perhaps toss in the towel on
> QMFv2 - then the final fix for this needs to take that plan into
> account.
>
> So I'm going to call a discussion among the developers, proposing we
> vote on what QMF support a 1.x C++ broker will provide.
I'd like to move the discussion to the user list as there are folks
there who may not be on dev list but would be impacted or might
otherwise have an opinion. I'm assuming everyone on the dev list is also
on the user list (if not you really should be :-)
By way of a short introduction, Ken's question comes after a long thread
focused on some issues around the dual support of v1 and v2 of the QMF
protocol in the console.py python library for applications built to
manage/monitor the c++ broker.
> Once that's decided, we can determine the most appropriate fix.
>
> As an aside - we've talked a lot about preserving behavior and
> functionality across 0.X releases in this thread. While nobody wants
> to gratuitously break stuff, this project has reserved the right to
> remove and change functionality between 0.x releases in order to fix
> bugs and provide new features. Things may be different once we get
> to 1.x, but for 0.x, this has been the case.
For my part, I am in favour of removing QMFv1 support, if necessary
providing some sort of adapter or plugin to deal with any use cases for
which this might prove problematic. I think QMFv2 is now firmly
established as the current management mechanism for the c++ broker.
Ultimately I also think we should be starting to think about an AMQP 1.0
based management standard and how we would transition to that. That is
however a larger question that shouldn't hold up the immediate decision
regarding QMFv1 though it may inform it a little.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Plans for QMF v1 and v2 (was Re: QMFv2 object update bug [WAS Re:
Questions from a novice])
Posted by Gordon Sim <gs...@redhat.com>.
On 05/03/2013 04:51 PM, Ken Giusti wrote:
> I want to know what the
> QPID project's plan is for QMFv1/v2 support in the C++ broker going
> forward.
>
> If the consensus is that a 1.x C++ broker is going to support both
> QMFv1 and QMFv2, then simply reverting all the changes I've made to
> qmf.console and reverting to the old behavior is probably the least
> risky move.
>
> On the other hand, if the consensus is to deprecate QMFv1 support
> prior to releasing a 1.0 C++ broker - or perhaps toss in the towel on
> QMFv2 - then the final fix for this needs to take that plan into
> account.
>
> So I'm going to call a discussion among the developers, proposing we
> vote on what QMF support a 1.x C++ broker will provide.
I'd like to move the discussion to the user list as there are folks
there who may not be on dev list but would be impacted or might
otherwise have an opinion. I'm assuming everyone on the dev list is also
on the user list (if not you really should be :-)
By way of a short introduction, Ken's question comes after a long thread
focused on some issues around the dual support of v1 and v2 of the QMF
protocol in the console.py python library for applications built to
manage/monitor the c++ broker.
> Once that's decided, we can determine the most appropriate fix.
>
> As an aside - we've talked a lot about preserving behavior and
> functionality across 0.X releases in this thread. While nobody wants
> to gratuitously break stuff, this project has reserved the right to
> remove and change functionality between 0.x releases in order to fix
> bugs and provide new features. Things may be different once we get
> to 1.x, but for 0.x, this has been the case.
For my part, I am in favour of removing QMFv1 support, if necessary
providing some sort of adapter or plugin to deal with any use cases for
which this might prove problematic. I think QMFv2 is now firmly
established as the current management mechanism for the c++ broker.
Ultimately I also think we should be starting to think about an AMQP 1.0
based management standard and how we would transition to that. That is
however a larger question that shouldn't hold up the immediate decision
regarding QMFv1 though it may inform it a little.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: QMFv2 object update bug [WAS Re: Questions from a novice]
Posted by Ken Giusti <kg...@redhat.com>.
Bill, Fraser,
Thanks for the well thought out replies.
I want to step back a bit.
Before I spend any more time on this problem, I want to know what the QPID project's plan is for QMFv1/v2 support in the C++ broker going forward.
If the consensus is that a 1.x C++ broker is going to support both QMFv1 and QMFv2, then simply reverting all the changes I've made to qmf.console and reverting to the old behavior is probably the least risky move.
On the other hand, if the consensus is to deprecate QMFv1 support prior to releasing a 1.0 C++ broker - or perhaps toss in the towel on QMFv2 - then the final fix for this needs to take that plan into account.
So I'm going to call a discussion among the developers, proposing we vote on what QMF support a 1.x C++ broker will provide.
Once that's decided, we can determine the most appropriate fix.
As an aside - we've talked a lot about preserving behavior and functionality across 0.X releases in this thread. While nobody wants to gratuitously break stuff, this project has reserved the right to remove and change functionality between 0.x releases in order to fix bugs and provide new features. Things may be different once we get to 1.x, but for 0.x, this has been the case.
-K
----- Original Message -----
> From: "Bill Freeman" <ke...@gmail.com>
> To: "Qpid Dev" <de...@qpid.apache.org>
> Sent: Monday, April 29, 2013 2:53:29 PM
> Subject: Fwd: QMFv2 object update bug [WAS Re: Questions from a novice]
>
> I messed up and only sent this tor Fraser originally, but intended it for
> all.
>
> Bill
>
> ---------- Forwarded message ----------
> From: Bill Freeman <ke...@gmail.com>
> Date: Mon, Apr 29, 2013 at 11:18 AM
> Subject: Re: QMFv2 object update bug [WAS Re: Questions from a novice]
> To: Fraser Adams <fr...@blueyonder.co.uk>
>
>
>
>
>
> On Sat, Apr 27, 2013 at 5:45 AM, Fraser Adams <fraser.adams@blueyonder.co.uk
> > wrote:
>
> > On 26/04/13 17:18, Ken Giusti wrote:
> >
> >> Hey Bill,
> >>
> >>
> >>
> >> I first started to implement the additional objectUpdate callback as
> >> originally proposed. Very easy to do. But that additional api required
> >> other tools to be updated - and additional documentation changes, etc.
> >>
> > Hmm I remain a little baffled here. So as I understood it prior to any
> > fixing going on here there existed a bug that actually prevented QMF2
> > updates actually hitting callbacks on the asynchronous python API, that's
> > good IMHO because most clients of that API almost certainly were expecting
> > a single callback call for a given object.
> >
> > I'm kind of unclear about "other tools to be updated" 'cause I'd have
> > assumed given the previous QMF2 bug that no existing tools were actually
> > receiving v2 updates so wouldn't need "fixed". Sure they will need modified
> > to handle V2 if they want to consume both, but I'd argue that it might be
> > better to modify tools that need to handle both than to risk breaking
> > changes which will blat an indeterminate number of third party tools. It's
> > not ideal for sure, but a non-breaking API extension requiring an addition
> > to clients to use it feels better and TBH it's actually clearer IMHO
> > because there's some conscious choice being made.
> >
> >
> >
> >> After implementing this, I discovered a problem with the fix: tools that
> >> need to monitor both QMFv1 and QMFv2 agents starting reporting incorrect
> >> results (qpid-tool, to be specific). These tools need to support both the
> >> old callbacks (for QMFv1 updates) and the new one (for QMFv2 updates), in
> >> order to see both QMFv1 and QMFv2 agents.
> >>
> > To be honest as I suggest above IMHO I actually think it's a good thing
> > for tools that wish to support both to be explicit. I think from the
> > various sections of this thread it's pretty clear that trying to crowbar
> > everything together is a *bad idea*. Sure QMF1 and QMF2 are essentially
> > representing equivalent information but there are sufficient differences
> > and subtleties to flash a great big red light. The reason it's good to have
> > a separate objectUpdate is because the QMF2 API *is* different at that
> > point because the protocol is sending what amounts to incompatible data.
> >
> > At that point there are two choices 1) an additional non-breaking API
> > callback or 2) a shim in the client side that makes things behave
> > identically.
> >
> > The latter may be a bad choice if both v1 and v2 updates arrive because
> > it'd need to apply a filter for the case of v1 plus v2 but it'd probably be
> > a good choice if it could select either v1 or v2 because then existing
> > clients wouldn't need to be modified at all.
> >
> >
> As long as I'm including the whole context, let me say that I think that
> Fraser is completely correct here. To paraphrase from a perhaps different
> point of view: Existing tools should work as they did before any of these
> un-released fixes were made. Having the previously undelivered v2 updates
> arrive at a callback that the existing tools didn't know about, and and,
> thus, for which they can't possibly be providing handlers, seems to fulfill
> this bill of particulars.
>
> One possible addition to make the old and new more similar would be to add
> a keyword argument to enable the first fix, meaning that tools which don't
> know bout that fix won't be consuming additional network bandwidth and
> compute cycles because of it. (I'm not suggesting this, since I don't see
> the additional traffic and processing as onerous, but if there are enough
> queues in someone else's circumstance, it could be relevant.) The
> corresponding change, a keyword to inhibit the reception of v1 updates, is
> at least as useful, but is riskier, or at least would require more thorough
> testing, since I'm not certain that the console whether tool itself
> depends upon information that I only knows how to extract from a v1
> response.
>
> >
> >
> >
> >
> >> And that's when I realized that this fix was just a hack to work around
> >> the actual problem. The real issue at hand is that a QMF agent must not
> >> transmit both QMFv1 and QMFv2 updates for the same object.
> >>
> >> That's really what's broken here, and the C++ broker is doing precisely
> >> that.
> >>
> > Whoa there :-) I think that the problem is that this is *exactly* what it
> > should be doing :-)
> >
> > OK what I mean is how does it know? Bear in mind that the updates causing
> > the fun here are *unsolicited* data pushes! These updates are being pushed
> > to a topic, so unless any client is subscribing to
> > qmf.default.topic/data.ind.# or whatever it isn't going to be getting v2
> > updates, similarly if it's not subscribing to the equivalent on
> > qpid.management/ it's not going to be getting v1 updates. OTOH if it's
> > subscribing to both that's exactly what it's getting.
> >
> > The key thing is there's a fairly large level of decoupling going on here,
> > so I don't believe that there's an especially easy way for the broker
> > ManagementAgent to work out who's asking for what and thus avoid
> > transmitting both v1 and v2.
> >
> >
> > Funnily enough in the v2 protocol and API there's actually no concept of
> > an *unsolicited* data push, so I'd argue that what the broker is currently
> > doing for the v2 updates is an "undocumented feature" :-), what V2 actually
> > talks about is the concept of "query subscriptions". With query
> > subscriptions the asynchronous pushes are actually *solicited* by the
> > client actually requesting some specified subset of the data, so the
> > Management Agent does actually have a mechanism to track what data any
> > given subscribers are actually interested in.
> >
>
> From the practical use case perspective, eliminating updates for queues
> which are not of interest, such as those involved in the implementation of
> QMF itself, sounds desirable (again, bandwidth and processing). On the
> other hand, I would have had to enumerate the queues I care about. Right
> now I'm using the cheap cop out that the names of all of "out" queues begin
> with a particular three character sequence.
>
> >
> > That's clearly a moot point, but it would help with this pickle.
> >
> >
> > I was musing over this problem and I believe that there is an option,
> > albeit a non-trivial one :-(
> >
> > So the *real* issue is that it's difficult to manage the carnage caused
> > when both v1 and v2 updates happen. You've previously suggested disabling
> > v1 by default, I've sort of got sympathy with that :-) However there are
> > two (I suspect significant) use cases that will be impacted by that choice:
> > 1) Any asynchronous console that relies on the existing v1 callbacks will
> > stop working - I suspect that in practice that'll mean every single darned
> > one of them except (at a push) qpid-tool. There may be a lot of third party
> > tools using this API so it may be an issue. At the very least there needs
> > to be some real good communication to the user list.
> > 2) If you disable v1 then any third party Agents will break too. Again
> > perhaps they need to change to v2 and this is a catalyst, but it just may
> > cause non-trivial business impact on users.
> >
> > Ultimately I think that it's very worth pushing out a survey on the user
> > list to gauge the potential impact.
> >
>
> I'd like to offer a caution about expecting this mailing list to inform
> users that a minor update (e.g.; 0.10 to 0.10.1 or even 0.11, as opposed
> to, for example, 0.10 to 1.0) will require using organizations to update
> custom tools. I essentially had my code working (with v1) before ever
> subscribing here (to learn what promises there might be about string
> representations of object IDs uniqueness). When this contract ends, I
> maybe back to doing Django, or *nix kernel internals, or even hardware
> development, and may not be able to justify the time to continue to read
> here. I'm sure that my circumstance isn't that unusual. So the client may
> no longer have someone following the list (let alone having invested the
> time to know what will impact them).
>
>
> > The non-trivial mitigation that I was musing over was whether there was a
> > possibility of constructing a v1->v2 bridge Agent? What I mean is that it
> > might be possible to get the broker ManagementAgent to only send v2 updates
> > if there's a mechanism for the broker to intercept v1 updates from third
> > party Agents and map those into v2 too, thus making every data push
> > consistently V2? I *think* this could work because I believe that v1
> > requires broker involvement in order to work (I think that was one of the
> > reasons for moving to v2).
> >
> > If it were possible to get to a clear point of only sending v2 updates
> > then it should be possible to have the asynchronous Console API shimmed in
> > such a way that it behaves *exactly* the same way with v2 updates as it did
> > with v1 updates so it's possible to mitigate the impact on third party
> > console applications (yay!).
> >
> > It has to be said It's be much better if users didn't have to rewrite
> > applications because of objectUpdate versus propUpdate/statUpdate but
> > clearly a shim would be required to make v2 updates behave correctly and
> > transparently for propUpdate/statUpdate callbacks.
> >
> > The result of the survey question "how many people have custom v1 Agents"
> > would help determine whether it's worth investing in a v1->v2 bridge Agent.
> >
> > A bridge ought to be possible, that's actually exactly what I wrote for
> > the Java Broker in the Qmf2ManagementAgent to get that to talk QMF2 (albeit
> > I was mapping from its internal model and not QMF1, but the principal is
> > the same - just don't ask me to do it in C++ though :-))
> >
> >
> >
> >
> >> Why is that a problem? Because a console receiving these two updates
> >> cannot tell that they are for the same object. To the console
> >> application,
> >> it appears as two separate objects.
> >>
> > As I mention above I don't believe that it's any easier for the broker to
> > do a "diff" given the unsolicited nature of these updates :-)
> >
> > I'm proposing that we should solve this problem by disabling QMFv1
> >> updates coming from the C++ broker. This should be the default for the
> >> next release (0.26). It should still be possible to turn them on manually
> >> if desired, but the C++ broker should only transmit QMFv2 type updates
> >> going forward.
> >>
> > As I say above I've got a fair amount of sympathy with this, and
> > parochially if it means that by doing this you can make statUpdate and
> > propUpdate for v2 behave *exactly* the same as for v1 so clients don't need
> > to be rewritten for v2 then it'd be better for me 'cause I only ever use v2
> > aside from this.
> >
> > However as I said above the only reason that this Console API ever
> > supported both was to cater for the case where a user needed to handle both
> > v1 and v2 updates and that use case is actually third party v1 Agents.
> >
> > I think it's important to at least check for how much impact this is going
> > to cause users.
> >
> >
> >
> >
> >> We can minimize the impact to console applications by having the
> >> objectProps and objectStats callbacks be invoked when a QMFv2 object
> >> arrives instead of introducing a new callback. [Bill - I'll fix the
> >> console
> >> to not call objectProps when only stats are present]. I think this should
> >> eliminate the need to re-code console applications.
> >>
> > So I don't think it's about "minimising the impact" I think that the API
> > needs to behave *exactly* the same for this to be any use. Basically any
> > asynchronous client hit by only v2 updates should behave identically to how
> > they behaved prior to this update or it's still introducing a regression
> > into an indeterminate number of applications. I've no idea how hard it'd be
> > to achieve this nirvana though.
> >
> >
> >
> >> This will cause problems for folks that use an old python console against
> >> the next release of the broker. These folks will have to upgrade the
> >> console as well.
> >>
> >> Does anyone have a better alternative?
> >>
> > Well I'm not necessarily saying that my suggestions above are *better* :-)
> > It's difficult for sure. I guess that my biggest concern is introducing
> > breaking changes against an unknown number of third party Consoles and
> > Agents.
> >
> > One other thing makes me nervous. I guess that there's the scenario where
> > users are working with a mixed economy of brokers at different versions or
> > who (like Bill) may be unable to alter the broker config to only push out a
> > particular version.
> >
> > All brokers from at least 0.8 can push out v2 so it *might* be safe to
> > have the console only subscribe to v2 updates (providing a switch to
> > specify v1). If you can make propUpdate and statUpdate behave consistently
> > then that only leaves the case where there are v1 Agents to support, which
> > is where the bridge Agent may come in.
> >
> > How much do you regret starting on this fix now Ken? :-D
> >
> >
> > It's worth pointing out that this is I suspect only the tip of the iceberg
> > for the sort of problems that might start to be encountered when migration
> > to AMQP 1.0 Management starts to gain traction. My strong hunch is that
> > bridge Agents/ManagementNodes are going to become pretty important friends
> > when it comes to trying to manage any transition. It's something that's
> > going to bite sooner or later and the C++, Java & Python communities are
> > going to have to be pretty joined up. I quite like QMF, but clearly some
> > mistakes were made along the way given the lack of support in the Java
> > community and frankly the fragmentation of APIs and lack of take up on the
> > "official" QMF2 API. We *really* need to try harder next time round and act
> > as an exemplar in the AMQP community.
> >
> > Frase
> >
> > Also, Ken previously mentioned, but Fraser didn't quote, that v1 and v2
> object identifiers are unrelated. I would like to point out that it is
> possible to synthesize the string representation of a v2 objectId using the
> information in a v1 objectProps() call. (After adding the first patch, and
> figuring out how to make things work again I was doing this to conjoin
> information from the v1 and v2 updates for a single queue.) The
> correspondences needed probably fall into the undocumented and subject to
> change category, but I suspect inertial will keep them from changing before
> 1.0.
>
> Bill
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Fwd: QMFv2 object update bug [WAS Re: Questions from a novice]
Posted by Bill Freeman <ke...@gmail.com>.
I messed up and only sent this tor Fraser originally, but intended it for
all.
Bill
---------- Forwarded message ----------
From: Bill Freeman <ke...@gmail.com>
Date: Mon, Apr 29, 2013 at 11:18 AM
Subject: Re: QMFv2 object update bug [WAS Re: Questions from a novice]
To: Fraser Adams <fr...@blueyonder.co.uk>
On Sat, Apr 27, 2013 at 5:45 AM, Fraser Adams <fraser.adams@blueyonder.co.uk
> wrote:
> On 26/04/13 17:18, Ken Giusti wrote:
>
>> Hey Bill,
>>
>>
>>
>> I first started to implement the additional objectUpdate callback as
>> originally proposed. Very easy to do. But that additional api required
>> other tools to be updated - and additional documentation changes, etc.
>>
> Hmm I remain a little baffled here. So as I understood it prior to any
> fixing going on here there existed a bug that actually prevented QMF2
> updates actually hitting callbacks on the asynchronous python API, that's
> good IMHO because most clients of that API almost certainly were expecting
> a single callback call for a given object.
>
> I'm kind of unclear about "other tools to be updated" 'cause I'd have
> assumed given the previous QMF2 bug that no existing tools were actually
> receiving v2 updates so wouldn't need "fixed". Sure they will need modified
> to handle V2 if they want to consume both, but I'd argue that it might be
> better to modify tools that need to handle both than to risk breaking
> changes which will blat an indeterminate number of third party tools. It's
> not ideal for sure, but a non-breaking API extension requiring an addition
> to clients to use it feels better and TBH it's actually clearer IMHO
> because there's some conscious choice being made.
>
>
>
>> After implementing this, I discovered a problem with the fix: tools that
>> need to monitor both QMFv1 and QMFv2 agents starting reporting incorrect
>> results (qpid-tool, to be specific). These tools need to support both the
>> old callbacks (for QMFv1 updates) and the new one (for QMFv2 updates), in
>> order to see both QMFv1 and QMFv2 agents.
>>
> To be honest as I suggest above IMHO I actually think it's a good thing
> for tools that wish to support both to be explicit. I think from the
> various sections of this thread it's pretty clear that trying to crowbar
> everything together is a *bad idea*. Sure QMF1 and QMF2 are essentially
> representing equivalent information but there are sufficient differences
> and subtleties to flash a great big red light. The reason it's good to have
> a separate objectUpdate is because the QMF2 API *is* different at that
> point because the protocol is sending what amounts to incompatible data.
>
> At that point there are two choices 1) an additional non-breaking API
> callback or 2) a shim in the client side that makes things behave
> identically.
>
> The latter may be a bad choice if both v1 and v2 updates arrive because
> it'd need to apply a filter for the case of v1 plus v2 but it'd probably be
> a good choice if it could select either v1 or v2 because then existing
> clients wouldn't need to be modified at all.
>
>
As long as I'm including the whole context, let me say that I think that
Fraser is completely correct here. To paraphrase from a perhaps different
point of view: Existing tools should work as they did before any of these
un-released fixes were made. Having the previously undelivered v2 updates
arrive at a callback that the existing tools didn't know about, and and,
thus, for which they can't possibly be providing handlers, seems to fulfill
this bill of particulars.
One possible addition to make the old and new more similar would be to add
a keyword argument to enable the first fix, meaning that tools which don't
know bout that fix won't be consuming additional network bandwidth and
compute cycles because of it. (I'm not suggesting this, since I don't see
the additional traffic and processing as onerous, but if there are enough
queues in someone else's circumstance, it could be relevant.) The
corresponding change, a keyword to inhibit the reception of v1 updates, is
at least as useful, but is riskier, or at least would require more thorough
testing, since I'm not certain that the console whether tool itself
depends upon information that I only knows how to extract from a v1
response.
>
>
>
>
>> And that's when I realized that this fix was just a hack to work around
>> the actual problem. The real issue at hand is that a QMF agent must not
>> transmit both QMFv1 and QMFv2 updates for the same object.
>>
>> That's really what's broken here, and the C++ broker is doing precisely
>> that.
>>
> Whoa there :-) I think that the problem is that this is *exactly* what it
> should be doing :-)
>
> OK what I mean is how does it know? Bear in mind that the updates causing
> the fun here are *unsolicited* data pushes! These updates are being pushed
> to a topic, so unless any client is subscribing to
> qmf.default.topic/data.ind.# or whatever it isn't going to be getting v2
> updates, similarly if it's not subscribing to the equivalent on
> qpid.management/ it's not going to be getting v1 updates. OTOH if it's
> subscribing to both that's exactly what it's getting.
>
> The key thing is there's a fairly large level of decoupling going on here,
> so I don't believe that there's an especially easy way for the broker
> ManagementAgent to work out who's asking for what and thus avoid
> transmitting both v1 and v2.
>
>
> Funnily enough in the v2 protocol and API there's actually no concept of
> an *unsolicited* data push, so I'd argue that what the broker is currently
> doing for the v2 updates is an "undocumented feature" :-), what V2 actually
> talks about is the concept of "query subscriptions". With query
> subscriptions the asynchronous pushes are actually *solicited* by the
> client actually requesting some specified subset of the data, so the
> Management Agent does actually have a mechanism to track what data any
> given subscribers are actually interested in.
>
>From the practical use case perspective, eliminating updates for queues
which are not of interest, such as those involved in the implementation of
QMF itself, sounds desirable (again, bandwidth and processing). On the
other hand, I would have had to enumerate the queues I care about. Right
now I'm using the cheap cop out that the names of all of "out" queues begin
with a particular three character sequence.
>
> That's clearly a moot point, but it would help with this pickle.
>
>
> I was musing over this problem and I believe that there is an option,
> albeit a non-trivial one :-(
>
> So the *real* issue is that it's difficult to manage the carnage caused
> when both v1 and v2 updates happen. You've previously suggested disabling
> v1 by default, I've sort of got sympathy with that :-) However there are
> two (I suspect significant) use cases that will be impacted by that choice:
> 1) Any asynchronous console that relies on the existing v1 callbacks will
> stop working - I suspect that in practice that'll mean every single darned
> one of them except (at a push) qpid-tool. There may be a lot of third party
> tools using this API so it may be an issue. At the very least there needs
> to be some real good communication to the user list.
> 2) If you disable v1 then any third party Agents will break too. Again
> perhaps they need to change to v2 and this is a catalyst, but it just may
> cause non-trivial business impact on users.
>
> Ultimately I think that it's very worth pushing out a survey on the user
> list to gauge the potential impact.
>
I'd like to offer a caution about expecting this mailing list to inform
users that a minor update (e.g.; 0.10 to 0.10.1 or even 0.11, as opposed
to, for example, 0.10 to 1.0) will require using organizations to update
custom tools. I essentially had my code working (with v1) before ever
subscribing here (to learn what promises there might be about string
representations of object IDs uniqueness). When this contract ends, I
maybe back to doing Django, or *nix kernel internals, or even hardware
development, and may not be able to justify the time to continue to read
here. I'm sure that my circumstance isn't that unusual. So the client may
no longer have someone following the list (let alone having invested the
time to know what will impact them).
> The non-trivial mitigation that I was musing over was whether there was a
> possibility of constructing a v1->v2 bridge Agent? What I mean is that it
> might be possible to get the broker ManagementAgent to only send v2 updates
> if there's a mechanism for the broker to intercept v1 updates from third
> party Agents and map those into v2 too, thus making every data push
> consistently V2? I *think* this could work because I believe that v1
> requires broker involvement in order to work (I think that was one of the
> reasons for moving to v2).
>
> If it were possible to get to a clear point of only sending v2 updates
> then it should be possible to have the asynchronous Console API shimmed in
> such a way that it behaves *exactly* the same way with v2 updates as it did
> with v1 updates so it's possible to mitigate the impact on third party
> console applications (yay!).
>
> It has to be said It's be much better if users didn't have to rewrite
> applications because of objectUpdate versus propUpdate/statUpdate but
> clearly a shim would be required to make v2 updates behave correctly and
> transparently for propUpdate/statUpdate callbacks.
>
> The result of the survey question "how many people have custom v1 Agents"
> would help determine whether it's worth investing in a v1->v2 bridge Agent.
>
> A bridge ought to be possible, that's actually exactly what I wrote for
> the Java Broker in the Qmf2ManagementAgent to get that to talk QMF2 (albeit
> I was mapping from its internal model and not QMF1, but the principal is
> the same - just don't ask me to do it in C++ though :-))
>
>
>
>
>> Why is that a problem? Because a console receiving these two updates
>> cannot tell that they are for the same object. To the console application,
>> it appears as two separate objects.
>>
> As I mention above I don't believe that it's any easier for the broker to
> do a "diff" given the unsolicited nature of these updates :-)
>
> I'm proposing that we should solve this problem by disabling QMFv1
>> updates coming from the C++ broker. This should be the default for the
>> next release (0.26). It should still be possible to turn them on manually
>> if desired, but the C++ broker should only transmit QMFv2 type updates
>> going forward.
>>
> As I say above I've got a fair amount of sympathy with this, and
> parochially if it means that by doing this you can make statUpdate and
> propUpdate for v2 behave *exactly* the same as for v1 so clients don't need
> to be rewritten for v2 then it'd be better for me 'cause I only ever use v2
> aside from this.
>
> However as I said above the only reason that this Console API ever
> supported both was to cater for the case where a user needed to handle both
> v1 and v2 updates and that use case is actually third party v1 Agents.
>
> I think it's important to at least check for how much impact this is going
> to cause users.
>
>
>
>
>> We can minimize the impact to console applications by having the
>> objectProps and objectStats callbacks be invoked when a QMFv2 object
>> arrives instead of introducing a new callback. [Bill - I'll fix the console
>> to not call objectProps when only stats are present]. I think this should
>> eliminate the need to re-code console applications.
>>
> So I don't think it's about "minimising the impact" I think that the API
> needs to behave *exactly* the same for this to be any use. Basically any
> asynchronous client hit by only v2 updates should behave identically to how
> they behaved prior to this update or it's still introducing a regression
> into an indeterminate number of applications. I've no idea how hard it'd be
> to achieve this nirvana though.
>
>
>
>> This will cause problems for folks that use an old python console against
>> the next release of the broker. These folks will have to upgrade the
>> console as well.
>>
>> Does anyone have a better alternative?
>>
> Well I'm not necessarily saying that my suggestions above are *better* :-)
> It's difficult for sure. I guess that my biggest concern is introducing
> breaking changes against an unknown number of third party Consoles and
> Agents.
>
> One other thing makes me nervous. I guess that there's the scenario where
> users are working with a mixed economy of brokers at different versions or
> who (like Bill) may be unable to alter the broker config to only push out a
> particular version.
>
> All brokers from at least 0.8 can push out v2 so it *might* be safe to
> have the console only subscribe to v2 updates (providing a switch to
> specify v1). If you can make propUpdate and statUpdate behave consistently
> then that only leaves the case where there are v1 Agents to support, which
> is where the bridge Agent may come in.
>
> How much do you regret starting on this fix now Ken? :-D
>
>
> It's worth pointing out that this is I suspect only the tip of the iceberg
> for the sort of problems that might start to be encountered when migration
> to AMQP 1.0 Management starts to gain traction. My strong hunch is that
> bridge Agents/ManagementNodes are going to become pretty important friends
> when it comes to trying to manage any transition. It's something that's
> going to bite sooner or later and the C++, Java & Python communities are
> going to have to be pretty joined up. I quite like QMF, but clearly some
> mistakes were made along the way given the lack of support in the Java
> community and frankly the fragmentation of APIs and lack of take up on the
> "official" QMF2 API. We *really* need to try harder next time round and act
> as an exemplar in the AMQP community.
>
> Frase
>
> Also, Ken previously mentioned, but Fraser didn't quote, that v1 and v2
object identifiers are unrelated. I would like to point out that it is
possible to synthesize the string representation of a v2 objectId using the
information in a v1 objectProps() call. (After adding the first patch, and
figuring out how to make things work again I was doing this to conjoin
information from the v1 and v2 updates for a single queue.) The
correspondences needed probably fall into the undocumented and subject to
change category, but I suspect inertial will keep them from changing before
1.0.
Bill
Re: QMFv2 object update bug [WAS Re: Questions from a novice]
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 26/04/13 17:18, Ken Giusti wrote:
> Hey Bill,
>
>
> I first started to implement the additional objectUpdate callback as originally proposed. Very easy to do. But that additional api required other tools to be updated - and additional documentation changes, etc.
Hmm I remain a little baffled here. So as I understood it prior to any
fixing going on here there existed a bug that actually prevented QMF2
updates actually hitting callbacks on the asynchronous python API,
that's good IMHO because most clients of that API almost certainly were
expecting a single callback call for a given object.
I'm kind of unclear about "other tools to be updated" 'cause I'd have
assumed given the previous QMF2 bug that no existing tools were actually
receiving v2 updates so wouldn't need "fixed". Sure they will need
modified to handle V2 if they want to consume both, but I'd argue that
it might be better to modify tools that need to handle both than to risk
breaking changes which will blat an indeterminate number of third party
tools. It's not ideal for sure, but a non-breaking API extension
requiring an addition to clients to use it feels better and TBH it's
actually clearer IMHO because there's some conscious choice being made.
>
> After implementing this, I discovered a problem with the fix: tools that need to monitor both QMFv1 and QMFv2 agents starting reporting incorrect results (qpid-tool, to be specific). These tools need to support both the old callbacks (for QMFv1 updates) and the new one (for QMFv2 updates), in order to see both QMFv1 and QMFv2 agents.
To be honest as I suggest above IMHO I actually think it's a good thing
for tools that wish to support both to be explicit. I think from the
various sections of this thread it's pretty clear that trying to crowbar
everything together is a *bad idea*. Sure QMF1 and QMF2 are essentially
representing equivalent information but there are sufficient differences
and subtleties to flash a great big red light. The reason it's good to
have a separate objectUpdate is because the QMF2 API *is* different at
that point because the protocol is sending what amounts to incompatible
data.
At that point there are two choices 1) an additional non-breaking API
callback or 2) a shim in the client side that makes things behave
identically.
The latter may be a bad choice if both v1 and v2 updates arrive because
it'd need to apply a filter for the case of v1 plus v2 but it'd probably
be a good choice if it could select either v1 or v2 because then
existing clients wouldn't need to be modified at all.
>
> And that's when I realized that this fix was just a hack to work around the actual problem. The real issue at hand is that a QMF agent must not transmit both QMFv1 and QMFv2 updates for the same object.
>
> That's really what's broken here, and the C++ broker is doing precisely that.
Whoa there :-) I think that the problem is that this is *exactly* what
it should be doing :-)
OK what I mean is how does it know? Bear in mind that the updates
causing the fun here are *unsolicited* data pushes! These updates are
being pushed to a topic, so unless any client is subscribing to
qmf.default.topic/data.ind.# or whatever it isn't going to be getting v2
updates, similarly if it's not subscribing to the equivalent on
qpid.management/ it's not going to be getting v1 updates. OTOH if it's
subscribing to both that's exactly what it's getting.
The key thing is there's a fairly large level of decoupling going on
here, so I don't believe that there's an especially easy way for the
broker ManagementAgent to work out who's asking for what and thus avoid
transmitting both v1 and v2.
Funnily enough in the v2 protocol and API there's actually no concept of
an *unsolicited* data push, so I'd argue that what the broker is
currently doing for the v2 updates is an "undocumented feature" :-),
what V2 actually talks about is the concept of "query subscriptions".
With query subscriptions the asynchronous pushes are actually
*solicited* by the client actually requesting some specified subset of
the data, so the Management Agent does actually have a mechanism to
track what data any given subscribers are actually interested in.
That's clearly a moot point, but it would help with this pickle.
I was musing over this problem and I believe that there is an option,
albeit a non-trivial one :-(
So the *real* issue is that it's difficult to manage the carnage caused
when both v1 and v2 updates happen. You've previously suggested
disabling v1 by default, I've sort of got sympathy with that :-) However
there are two (I suspect significant) use cases that will be impacted by
that choice:
1) Any asynchronous console that relies on the existing v1 callbacks
will stop working - I suspect that in practice that'll mean every single
darned one of them except (at a push) qpid-tool. There may be a lot of
third party tools using this API so it may be an issue. At the very
least there needs to be some real good communication to the user list.
2) If you disable v1 then any third party Agents will break too. Again
perhaps they need to change to v2 and this is a catalyst, but it just
may cause non-trivial business impact on users.
Ultimately I think that it's very worth pushing out a survey on the user
list to gauge the potential impact.
The non-trivial mitigation that I was musing over was whether there was
a possibility of constructing a v1->v2 bridge Agent? What I mean is that
it might be possible to get the broker ManagementAgent to only send v2
updates if there's a mechanism for the broker to intercept v1 updates
from third party Agents and map those into v2 too, thus making every
data push consistently V2? I *think* this could work because I believe
that v1 requires broker involvement in order to work (I think that was
one of the reasons for moving to v2).
If it were possible to get to a clear point of only sending v2 updates
then it should be possible to have the asynchronous Console API shimmed
in such a way that it behaves *exactly* the same way with v2 updates as
it did with v1 updates so it's possible to mitigate the impact on third
party console applications (yay!).
It has to be said It's be much better if users didn't have to rewrite
applications because of objectUpdate versus propUpdate/statUpdate but
clearly a shim would be required to make v2 updates behave correctly and
transparently for propUpdate/statUpdate callbacks.
The result of the survey question "how many people have custom v1
Agents" would help determine whether it's worth investing in a v1->v2
bridge Agent.
A bridge ought to be possible, that's actually exactly what I wrote for
the Java Broker in the Qmf2ManagementAgent to get that to talk QMF2
(albeit I was mapping from its internal model and not QMF1, but the
principal is the same - just don't ask me to do it in C++ though :-))
>
> Why is that a problem? Because a console receiving these two updates cannot tell that they are for the same object. To the console application, it appears as two separate objects.
As I mention above I don't believe that it's any easier for the broker
to do a "diff" given the unsolicited nature of these updates :-)
> I'm proposing that we should solve this problem by disabling QMFv1 updates coming from the C++ broker. This should be the default for the next release (0.26). It should still be possible to turn them on manually if desired, but the C++ broker should only transmit QMFv2 type updates going forward.
As I say above I've got a fair amount of sympathy with this, and
parochially if it means that by doing this you can make statUpdate and
propUpdate for v2 behave *exactly* the same as for v1 so clients don't
need to be rewritten for v2 then it'd be better for me 'cause I only
ever use v2 aside from this.
However as I said above the only reason that this Console API ever
supported both was to cater for the case where a user needed to handle
both v1 and v2 updates and that use case is actually third party v1 Agents.
I think it's important to at least check for how much impact this is
going to cause users.
>
> We can minimize the impact to console applications by having the objectProps and objectStats callbacks be invoked when a QMFv2 object arrives instead of introducing a new callback. [Bill - I'll fix the console to not call objectProps when only stats are present]. I think this should eliminate the need to re-code console applications.
So I don't think it's about "minimising the impact" I think that the API
needs to behave *exactly* the same for this to be any use. Basically any
asynchronous client hit by only v2 updates should behave identically to
how they behaved prior to this update or it's still introducing a
regression into an indeterminate number of applications. I've no idea
how hard it'd be to achieve this nirvana though.
>
> This will cause problems for folks that use an old python console against the next release of the broker. These folks will have to upgrade the console as well.
>
> Does anyone have a better alternative?
Well I'm not necessarily saying that my suggestions above are *better*
:-) It's difficult for sure. I guess that my biggest concern is
introducing breaking changes against an unknown number of third party
Consoles and Agents.
One other thing makes me nervous. I guess that there's the scenario
where users are working with a mixed economy of brokers at different
versions or who (like Bill) may be unable to alter the broker config to
only push out a particular version.
All brokers from at least 0.8 can push out v2 so it *might* be safe to
have the console only subscribe to v2 updates (providing a switch to
specify v1). If you can make propUpdate and statUpdate behave
consistently then that only leaves the case where there are v1 Agents to
support, which is where the bridge Agent may come in.
How much do you regret starting on this fix now Ken? :-D
It's worth pointing out that this is I suspect only the tip of the
iceberg for the sort of problems that might start to be encountered when
migration to AMQP 1.0 Management starts to gain traction. My strong
hunch is that bridge Agents/ManagementNodes are going to become pretty
important friends when it comes to trying to manage any transition. It's
something that's going to bite sooner or later and the C++, Java &
Python communities are going to have to be pretty joined up. I quite
like QMF, but clearly some mistakes were made along the way given the
lack of support in the Java community and frankly the fragmentation of
APIs and lack of take up on the "official" QMF2 API. We *really* need to
try harder next time round and act as an exemplar in the AMQP community.
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: QMFv2 object update bug [WAS Re: Questions from a novice]
Posted by Ken Giusti <kg...@redhat.com>.
Hey Bill,
I've cc'ing Fraser and the rest of the dev team on this - been meaning to raise this as a general issue with the C++ broker.
I'm trying to find the best way to fix these problems, and I've started making changes on trunk. You've been bitten by my latest attempts - sorry.
I first started to implement the additional objectUpdate callback as originally proposed. Very easy to do. But that additional api required other tools to be updated - and additional documentation changes, etc.
After implementing this, I discovered a problem with the fix: tools that need to monitor both QMFv1 and QMFv2 agents starting reporting incorrect results (qpid-tool, to be specific). These tools need to support both the old callbacks (for QMFv1 updates) and the new one (for QMFv2 updates), in order to see both QMFv1 and QMFv2 agents.
And that's when I realized that this fix was just a hack to work around the actual problem. The real issue at hand is that a QMF agent must not transmit both QMFv1 and QMFv2 updates for the same object.
That's really what's broken here, and the C++ broker is doing precisely that.
Why is that a problem? Because a console receiving these two updates cannot tell that they are for the same object. To the console application, it appears as two separate objects.
Yep - QMFv1 object identifiers have no relation to QMFv2 object identifiers, by design. Any attempt to compare the two will result in inequality, even if the underlying object is the same.
Take a look at qpid-tool on trunk. Run it against a broker. It will end up reporting twice as many queues, exchanges, etc, and there really are. This is because the tool stores the objects by their object ids - which is precisely what it should do. But the broker is sending two updates for each object, and now that the console is no longer dropping the QMFv2 update, its adding those objects to its database. Since the object ids for the two updates don't match, the database stores each as a separate object.
I'm proposing that we should solve this problem by disabling QMFv1 updates coming from the C++ broker. This should be the default for the next release (0.26). It should still be possible to turn them on manually if desired, but the C++ broker should only transmit QMFv2 type updates going forward.
We can minimize the impact to console applications by having the objectProps and objectStats callbacks be invoked when a QMFv2 object arrives instead of introducing a new callback. [Bill - I'll fix the console to not call objectProps when only stats are present]. I think this should eliminate the need to re-code console applications.
This will cause problems for folks that use an old python console against the next release of the broker. These folks will have to upgrade the console as well.
Does anyone have a better alternative?
-K
----- Original Message -----
> From: "Bill Freeman" <ke...@gmail.com>
> To: "Ken Giusti" <kg...@redhat.com>
> Sent: Thursday, April 25, 2013 5:07:44 PM
> Subject: Re: QMFv2 object update bug [WAS Re: Questions from a novice]
>
> On Wed, Apr 10, 2013 at 11:41 AM, Ken Giusti <kg...@redhat.com> wrote:
>
> > Folks,
> >
> > Recently I submitted a fix on trunk that addressed a problem where QMF v2
> > object update notifications were being dropped by the python QMF console
> > (qmf.console). The fix merely re-enabled the update notifications.
> >
> > As pointed out by both Fraser Adams and Bill, this may cause problems with
> > existing clients. As described below (and elsewhere in this thread), now
> > both QMFv1 updates and QMFv2 updates will be delivered via the
> > qmf.console.Console callback interface.
> >
> > Sadly, the way this is done is confusing at best. For V1 updates, there
> > are separate callbacks for statistics updates (via objectStats()) and
> > property updates (via objectProps). The confusion comes from the fact that
> > V2 handles these updates in a different way: it is optimized to generate
> > only one update that combines both statistics and properties. However that
> > update arrives via the objectProps() callback. For QMFv2, the
> > objectStats() callback is ignored. This forces the application's
> > objectProps to have to deal with different update formats - v1 and v2 - and
> > handle the different use cases (updating statistics in the v2 case, not for
> > v1).
> >
> > AFAIK - this bug has prevented v2 updates from arriving on that
> > objectProps callback since prior to 0.18. Its likely that apps written
> > using the console haven't ever received v2 style updates.
> >
> > I propose we fix this in the manner suggested by Bill and Fraser.
> > Specifically:
> >
> > 1) Add a new interface to the qmf.console.Console object:
> > def objectUpdate( self, broker, record )
> > This callback will be invoked only for QMFv2 object updates. The
> > parameters are analogous to objectProps and objectStats
> >
> > 2) The existing objectProps and objectStats callbacks are invoked only for
> > the v1 style updates, as was their behavior prior to my fix. v2 object
> > updates will never arrive on these callbacks.
> >
> > Any objections?
> >
> >
> I may be cross eyed, but it appears to me, in the latest in subversion
> checkout, that something different happened. (I'm getting ready to deploy,
> and want to make something I can install with pip, so I'm looking to work
> with the latest and use v2, so that led me to poke in subversion.)
>
> Rather than v2 updates calling session.concole.objectUpdate() only, there
> is still no session.console.objectUpdate() call, and a v2 update that
> contains statistics will also call both session.console.objectProps() and
> session.console.objectStats(). See around line 3112.
>
> (As to subsequent updates, they might only go to objectStats(), though I
> doubt it, since, after the patch in _tryToConnect(), I never saw a v2
> update call objectStats(), only objectProps(), so I'm guessing that v2
> updates always have qmf_object in the keywords, and never qmf_object_stats
> .)
>
> With this approach objectProps() will still be getting both v1 and v2
> calls, and will sometimes get v2 calls without properties. I believe that
> any unknown existing code out there is still likely to break if they
> upgrade to this qmf.console. That would not be true with Ken's suggested
> approach above. Old code, never having heard of the objectUpdate() console
> callback method, won't have a handler for it, and the v2 callbacks would be
> ignored, rather than causing trouble.
>
> Was this, perchance, because there is within qmf.console.Agent.__call__,
> some expense in determining whether an update is v1 or v2 (instantiating
> the ObjectID)? Because I'm going to wind up doing that anyway, since I
> need to discard one set of updates or the other.
>
> (Isn't anyone using this API going to need the ObjectId instance anyway, to
> be able to figure out to which object this update applies? If it's
> expensive to instantiate the ObjectId, then it's actually a pity that
> Agent.__call__ doesn't always do it and pass it to callbacks, allowing it
> to only be done once per update.)
>
> Bill
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
QMFv2 object update bug [WAS Re: Questions from a novice]
Posted by Ken Giusti <kg...@redhat.com>.
Folks,
Recently I submitted a fix on trunk that addressed a problem where QMF v2 object update notifications were being dropped by the python QMF console (qmf.console). The fix merely re-enabled the update notifications.
As pointed out by both Fraser Adams and Bill, this may cause problems with existing clients. As described below (and elsewhere in this thread), now both QMFv1 updates and QMFv2 updates will be delivered via the qmf.console.Console callback interface.
Sadly, the way this is done is confusing at best. For V1 updates, there are separate callbacks for statistics updates (via objectStats()) and property updates (via objectProps). The confusion comes from the fact that V2 handles these updates in a different way: it is optimized to generate only one update that combines both statistics and properties. However that update arrives via the objectProps() callback. For QMFv2, the objectStats() callback is ignored. This forces the application's objectProps to have to deal with different update formats - v1 and v2 - and handle the different use cases (updating statistics in the v2 case, not for v1).
AFAIK - this bug has prevented v2 updates from arriving on that objectProps callback since prior to 0.18. Its likely that apps written using the console haven't ever received v2 style updates.
I propose we fix this in the manner suggested by Bill and Fraser. Specifically:
1) Add a new interface to the qmf.console.Console object:
def objectUpdate( self, broker, record )
This callback will be invoked only for QMFv2 object updates. The parameters are analogous to objectProps and objectStats
2) The existing objectProps and objectStats callbacks are invoked only for the v1 style updates, as was their behavior prior to my fix. v2 object updates will never arrive on these callbacks.
Any objections?
----- Original Message -----
> From: "Bill Freeman" <ke...@gmail.com>
> To: users@qpid.apache.org
> Sent: Friday, April 5, 2013 4:20:37 PM
> Subject: Re: Questions from a novice
>
> On Fri, Apr 5, 2013 at 3:38 PM, Ken Giusti <kg...@redhat.com> wrote:
>
> > ----- Original Message -----
> > > From: "Bill Freeman" <ke...@gmail.com>
> >
> > QMF V2 has been available for some time now, and we've cut all the project
> > tools and libraries over to speaking it. If we start down the road to
> > deprecating V1, turning off the broker updates is about the least risky
> > first step I can think of. It would not remove v1 at all - but it would
> > require manual intervention to restore the old behavior. Having to
> > explicitly take that action would be a (hopefully gentle) nudge to assess a
> > deployment's V1 dependency.
> >
> >
> > I think the on line documentation may have something to answer for here.
> When I started this project, since we are using MRG, I started with
> RedHat's documentation, but didn't find any advice about coding QMF. Then
> Google got me to the Apache Qpid pages, link to QMF, and eventually
> dribbled down to an example that became the base of my current design.
> That was based on qmf.console. There was no insight there about selecting
> V1 versus V2. There were also several pages, at least, that seemed to
> imply that V1 was now, and everything would support it, while V2 was new
> and for the future, and not universally supported (and that turned out to
> be correct until your patch).
>
> I think that the concept that consoles, for some time yet to come, should
> endeavor to support both versions, is a good one. They should perhaps log
> a deprecation warning if they discover a V1 only broker (aren't V2 brokers
> supposed to translate for V1 only agents?), but they should work anyway.
>
> One thing that makes it tough for consoles to do so, at least if they are
> based on qmf.console, is the fact that the objectProps callback is shared
> between versions, but has a different meaning. One callback carrying
> deltas is a good design, but it seems that either it should use a
> completely separate callback, or it should call both objectProps and
> objectStats (in that order), omitting objectProps if it has no properties,
> and omitting objectStats if it has no statistics. A V2 specific callback
> would be my preference, but only by a little. Since the other stuff has
> never been called for V2 updates before, this is the time to make that
> change.
>
> And I guess that's also a potential problem with making the fix from your
> patch: Code that has been working for years using qmf.console has never
> received a V2 objectProps call before may not work with it (mine certainly
> didn't). A separate callback for V2 updates fixes this issue.
>
> Bill
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
On Fri, Apr 5, 2013 at 3:38 PM, Ken Giusti <kg...@redhat.com> wrote:
> ----- Original Message -----
> > From: "Bill Freeman" <ke...@gmail.com>
>
> QMF V2 has been available for some time now, and we've cut all the project
> tools and libraries over to speaking it. If we start down the road to
> deprecating V1, turning off the broker updates is about the least risky
> first step I can think of. It would not remove v1 at all - but it would
> require manual intervention to restore the old behavior. Having to
> explicitly take that action would be a (hopefully gentle) nudge to assess a
> deployment's V1 dependency.
>
>
> I think the on line documentation may have something to answer for here.
When I started this project, since we are using MRG, I started with
RedHat's documentation, but didn't find any advice about coding QMF. Then
Google got me to the Apache Qpid pages, link to QMF, and eventually
dribbled down to an example that became the base of my current design.
That was based on qmf.console. There was no insight there about selecting
V1 versus V2. There were also several pages, at least, that seemed to
imply that V1 was now, and everything would support it, while V2 was new
and for the future, and not universally supported (and that turned out to
be correct until your patch).
I think that the concept that consoles, for some time yet to come, should
endeavor to support both versions, is a good one. They should perhaps log
a deprecation warning if they discover a V1 only broker (aren't V2 brokers
supposed to translate for V1 only agents?), but they should work anyway.
One thing that makes it tough for consoles to do so, at least if they are
based on qmf.console, is the fact that the objectProps callback is shared
between versions, but has a different meaning. One callback carrying
deltas is a good design, but it seems that either it should use a
completely separate callback, or it should call both objectProps and
objectStats (in that order), omitting objectProps if it has no properties,
and omitting objectStats if it has no statistics. A V2 specific callback
would be my preference, but only by a little. Since the other stuff has
never been called for V2 updates before, this is the time to make that
change.
And I guess that's also a potential problem with making the fix from your
patch: Code that has been working for years using qmf.console has never
received a V2 objectProps call before may not work with it (mine certainly
didn't). A separate callback for V2 updates fixes this issue.
Bill
Re: Questions from a novice
Posted by Ken Giusti <kg...@redhat.com>.
----- Original Message -----
> From: "Bill Freeman" <ke...@gmail.com>
> To: users@qpid.apache.org
> Sent: Friday, April 5, 2013 2:37:23 PM
> Subject: Re: Questions from a novice
>
> On Fri, Apr 5, 2013 at 2:09 PM, Ken Giusti <kg...@redhat.com> wrote:
>
> >
> >
> > ----- Original Message -----
> > > From: "Bill Freeman" <ke...@gmail.com>
> > ...
> > >
> > > However, it would be nice to know how to get the console to only
> > subscribe
> > > to the V2 queues. It's not clear that I can require the brokers to be
> > > reconfigured as no v1, so it would be nice if there were a console
> > option.
> > > I'll do a bit more reading of the source code, but if anyone can shortcut
> > > that, I'd appreciate it.
> > >
> >
> > Hmmmm... you know, I don't think the console allows you to filter out v1.
> >
>
> If it has already arrived in the console process, then the bandwidth has
> already been used, and it makes little difference whether I filter it or
> the underlying console code does. I'd been thinking that, if at the time
> the session is set up, the console did not subscribe to the V1 queues (or
> should I say bind to the V1 exchanges?), then V1 updates wouldn't get sent
> over the wire. (In fact, I have half a memory somewhere that if nobody
> subscribes to the queues, the broker saves its own resources and time by
> not issuing the updates.)
>
Yes, absolutely true and very astute - simply dropping the update at the console would be wasteful. My own very bad choice of words - the actual 'filter' would need to work exactly as you describe: when set, prevent the console from binding to the V1 exchanges.
Using such a option would not be without some risk to those deployments that have old V1 agents attached to the broker: you'd no longer get V1 updates from the broker, but you'd also lose access to all V1 agents attached to that broker.
>
> >
> > Personally, I'd like to see the default behavior of the broker changed to
> > only send V2 updates, rather than both. People who need v1 would have the
> > option of turning it on. Too late to ask for that for the next release,
> > but I'll add a JIRA and try to get that into the following release. Or at
> > least add an option to filter v1 at the console.
> >
>
> There is the hazard that someone using some V1 tools will have a hard time
> deciding why, after an upgrade, things have stopped working. After all,
> the broker still supports V1 (if their knowledge is that deep). I'm only a
> contractor where I'm doing this, and I might easily have finished the
> project with V1. It's not clear who will understand deeply enough after
> I've moved on.
>
> Bill
>
Very true, but at some point I imagine that V1 will be deprecated. There is a cost, both in person-power and code complexity, to maintaining two parallel management implementations. Heck, I just spent two days fixing a bug that seems to be related to a V1 convention that broke the V2 path. And since the tests speak both variants, we didn't hit the bug as the functioning V1 code "hid" the V2 problem.
QMF V2 has been available for some time now, and we've cut all the project tools and libraries over to speaking it. If we start down the road to deprecating V1, turning off the broker updates is about the least risky first step I can think of. It would not remove v1 at all - but it would require manual intervention to restore the old behavior. Having to explicitly take that action would be a (hopefully gentle) nudge to assess a deployment's V1 dependency.
Of course, this is just my $0.02, certainly not a consensus by any means :)
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
On Fri, Apr 5, 2013 at 2:37 PM, Bill Freeman <ke...@gmail.com> wrote:
> On Fri, Apr 5, 2013 at 2:09 PM, Ken Giusti <kg...@redhat.com> wrote:
>
>> ----- Original Message -----
>> > From: "Bill Freeman" <ke...@gmail.com>
>> ...
>> >
>> > However, it would be nice to know how to get the console to only
>> subscribe
>> > to the V2 queues. It's not clear that I can require the brokers to be
>> > reconfigured as no v1, so it would be nice if there were a console
>> option.
>> > I'll do a bit more reading of the source code, but if anyone can
>> shortcut
>> > that, I'd appreciate it.
>> >
>>
>> Hmmmm... you know, I don't think the console allows you to filter out v1.
>>
>
> If it has already arrived in the console process, then the bandwidth has
> already been used, and it makes little difference whether I filter it or
> the underlying console code does. I'd been thinking that, if at the time
> the session is set up, the console did not subscribe to the V1 queues (or
> should I say bind to the V1 exchanges?), then V1 updates wouldn't get sent
> over the wire. (In fact, I have half a memory somewhere that if nobody
> subscribes to the queues, the broker saves its own resources and time by
> not issuing the updates.)
>
>
It looks like the heart of not getting V1 stuff is in
qmf.console.Broker._tryToConnect(). If a flag could get passed in (say,
noV1=True in the connectArguments, which the Broker.__init__() could pop
and save, with a default of False), then _tryToConnect() could omit all the
"rdest" and "tdest" setup, leaving only the V2 stuff (noV1 and not
brokerSupportsV2 would need to raise an error).
Of course there might be other places that specifically try to do V1 stuff
that I'm unaware of (e.g.; fetching a schema), and doesn't pay attention to
brokerSupportsV2. And I'm looking around in code that I don't fully
understand. And I can't find that doc that said that updates aren't
generated without a subscription, and this might not be separately
calculated for V1 and V2.
Bill
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
On Fri, Apr 5, 2013 at 2:09 PM, Ken Giusti <kg...@redhat.com> wrote:
>
>
> ----- Original Message -----
> > From: "Bill Freeman" <ke...@gmail.com>
> ...
> >
> > However, it would be nice to know how to get the console to only
> subscribe
> > to the V2 queues. It's not clear that I can require the brokers to be
> > reconfigured as no v1, so it would be nice if there were a console
> option.
> > I'll do a bit more reading of the source code, but if anyone can shortcut
> > that, I'd appreciate it.
> >
>
> Hmmmm... you know, I don't think the console allows you to filter out v1.
>
If it has already arrived in the console process, then the bandwidth has
already been used, and it makes little difference whether I filter it or
the underlying console code does. I'd been thinking that, if at the time
the session is set up, the console did not subscribe to the V1 queues (or
should I say bind to the V1 exchanges?), then V1 updates wouldn't get sent
over the wire. (In fact, I have half a memory somewhere that if nobody
subscribes to the queues, the broker saves its own resources and time by
not issuing the updates.)
>
> Personally, I'd like to see the default behavior of the broker changed to
> only send V2 updates, rather than both. People who need v1 would have the
> option of turning it on. Too late to ask for that for the next release,
> but I'll add a JIRA and try to get that into the following release. Or at
> least add an option to filter v1 at the console.
>
There is the hazard that someone using some V1 tools will have a hard time
deciding why, after an upgrade, things have stopped working. After all,
the broker still supports V1 (if their knowledge is that deep). I'm only a
contractor where I'm doing this, and I might easily have finished the
project with V1. It's not clear who will understand deeply enough after
I've moved on.
Bill
Re: Questions from a novice
Posted by Ken Giusti <kg...@redhat.com>.
----- Original Message -----
> From: "Bill Freeman" <ke...@gmail.com>
> To: users@qpid.apache.org
> Sent: Friday, April 5, 2013 12:45:02 PM
> Subject: Re: Questions from a novice
>
> Ken,
>
> Yes, with the patch in, I get callbacks to my qmf.console.Console sub-class
> instance's callback methods for both V1 and V2 updates.
Great! I've submitted this fix to QPID. If it proves stable, I'll ask permission to get it in our next release candidate.
>
> There are some differences in the plumbing:
>
> V1 initial updates carry properties only. V2 initial updates carry
> both properties and statistics. Both are sent to the objectProps()
> callback.
>
> Subsequent updates carry only statistics, for both V1 and V2. However,
> for V1 these updates are sent to the objectStats() callback, while for V2
> they are sent to the objectProps() callback. (I wonder if this was
> intended, or whether these were intended to go to objectStats() in V2 as
> well).
I have to smile at this - when I was fixing the bug I was surprised at this behavior. I had to look into a bit but, yes, it actually is intended (though it certainly needs more documentation). A bit of a history lesson as to why it works this way: Pre-V2, we defined the Console callback interface. At that time (Qmfv1), object statistic updates and object property updates were sent in separate messages. This was done as an optimization - in V1 all update messages are statically defined: they contain all properties (for property update) or all statistics (for statistic updates). If we didn't split along property/statistics - we'd have to send the entire object every time there was a statistics update. So there were two types of update messages, and hence two types of callbacks.
Now along comes V2, which allows for "sparse" object updates (you only have to send those properties/statistics that change - not all of them). So now we only send one object update message, which contains only those properties and stats that have changed. Much more efficient. But what to do about the existing API? We probably should've broken backward compatibility, and I don't know the details (or, more likely, have blocked them out) but instead we only call objectProps() for these updates.
>
> Hereafter "oid" is an instance of qmf.console.ObjectId acquired by calling
> the getObjectId() method of the qmf.console.Object instance that is passed
> to the callbacks.
>
> Yes, str(oid) is much prettier for V2. Having the final component be the
> queue name clearly assures stability across broker restarts and unrelated
> configurations. I had trouble with that belief for the hash that is the
> final component of the same string for V1.
>
> But the first 4 (dash separated) components of str(oid) don't seem useful
> for my purpose of a within broker unique and restart stable identifier.
> (Those first four come from oid.getFlags(), oid.getSequence(),
> oid.getBrokerBank(), and oid.getAgentBank(), and turn into single digit
> numbers for my test case.) I think that oid.getObject() (returns
> oid.objectName, using this directly might be clearer) is more than enough
> (on a V2 ObjectId instance, anyway). Please let me know if you disagree.
>
Yes, you should be fine using oid.getObject(). The other stuff, flags, broker bank, etc, are probably irrelevant for your purposed. The Sequence number will increment on each restart of qpidd for non-durable objects (non-durable queues, exchanges, etc), but what that really means is up to your application. If you're planning on using the same object names across broker restarts to identify 'logically' same object, you can ignore sequence number changes.
> oid.isV2 seems to be the correct place to check to know which sort of
> record I have. For now, it I only want to process one kind of record (the
> useful V1 and V2 information is redundant) I'm going to use this flag.
Yes, that flag is used to distinguish between the two formats.
>
> However, it would be nice to know how to get the console to only subscribe
> to the V2 queues. It's not clear that I can require the brokers to be
> reconfigured as no v1, so it would be nice if there were a console option.
> I'll do a bit more reading of the source code, but if anyone can shortcut
> that, I'd appreciate it.
>
Hmmmm... you know, I don't think the console allows you to filter out v1.
Personally, I'd like to see the default behavior of the broker changed to only send V2 updates, rather than both. People who need v1 would have the option of turning it on. Too late to ask for that for the next release, but I'll add a JIRA and try to get that into the following release. Or at least add an option to filter v1 at the console.
> Bill
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Hi Guys,
This thread has intrigued me a bit.
I must admit that I had thought the qmf.console.Console,
qmf.console.Session stuff was QMF1 only so I was intrigued by the
comments about QMF2. I dug out an old python program.
It starts like:
class AuditConsole(Console):
# AuditConsole Constructor
def __init__(self, whitelistFile):
self.whitelistFile = whitelistFile
self.whitelistLastModified = None
# Declare dictionary to map queue name to queue objects
self.queueNameMap = {}
# Declare dictionary to map queue name to subscription objects
......
so it's subclassing Console and using the asynchronous approach. I
initialise it with:
console = AuditConsole(options.whitelist)
session = Session(console, rcvObjects=True, rcvEvents=True,
userBindings=True, manageConnections=True)
session.bindClass("org.apache.qpid.broker", "queue")
session.bindClass("org.apache.qpid.broker", "exchange")
session.bindClass("org.apache.qpid.broker", "binding")
session.bindClass("org.apache.qpid.broker", "connection")
session.bindClass("org.apache.qpid.broker", "session")
session.bindClass("org.apache.qpid.broker", "subscription")
brokers = []
try:
try:
for host in arguments:
brokers.append(session.addBroker(host, None,
options.sasl_mechanism))
I've got code
def objectStats(self, broker, record):
classKey = record.getClassKey()
oid = record.getObjectId()
# Note that queue objects are keyed by both queue name and
Object ID
if classKey.getClassName() == "queue":
print "queue: %s" % (oid.asMap())
def objectProps(self, broker, record):
classKey = record.getClassKey()
oid = record.getObjectId()
# Note that queue objects are keyed by both queue name and
Object ID
if classKey.getClassName() == "queue":
print "queue: %s %s" % (record.name, oid.asMap())
but I only seem to be getting QMF1 push updates given the ObjectId
queue: qmfc-v2-hb-zappa.4505.1 {'_object_name': '25', '_agent_name':
'0', '_agent_epoch': 1662}
queue: {'_object_name': '25', '_agent_name': '0', '_agent_epoch': 1662}
queue: qmfc-v2-ui-zappa.4505.1 {'_object_name': '23', '_agent_name':
'0', '_agent_epoch': 1662}
queue: {'_object_name': '23', '_agent_name': '0', '_agent_epoch': 1662}
queue: qmfc-v2-zappa.4505.1 {'_object_name': '21', '_agent_name': '0',
'_agent_epoch': 1662}
queue: {'_object_name': '21', '_agent_name': '0', '_agent_epoch': 1662}
queue: reply-zappa.4505.1 {'_object_name': '14', '_agent_name': '0',
'_agent_epoch': 1662}
queue: {'_object_name': '14', '_agent_name': '0', '_agent_epoch': 1662}
queue: topic-zappa.4505.1 {'_object_name': '18', '_agent_name': '0',
'_agent_epoch': 1662}
queue: {'_object_name': '18', '_agent_name': '0', '_agent_epoch': 1662}
queue: {'_object_name': '23', '_agent_name': '0', '_agent_epoch': 1662}
queue: {'_object_name': '18', '_agent_name': '0', '_agent_epoch': 1662}
queue: {'_object_name': '23', '_agent_name': '0', '_agent_epoch': 1662}
queue: {'_object_name': '18', '_agent_name': '0', '_agent_epoch': 1662}
I'm running qpidd 0.23 with no options so both QMF1 and QMF2 data push
should be available.
What's even odder is that clearly object name 23 relates to
qmfc-v2-ui-zappa.4505.1 which is, given the name, one of the QMF2 queues.
I'm not at all sure why I should be seeing V1 ObjectIds
I'm not particularly familiar with this API (nor really with python).
The impression I've got from Bill's last post is that he's seeing both
the QMF1 and QMF2 updates (that seems wrong anyway, I don't think being
notified for each QMF version is what anyone would expect) but I only
seem to be seeing QMF1 ObjectIds albeit relating to what appears to be
QMF2 queues??????
Any thoughts??
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
Ken,
Yes, with the patch in, I get callbacks to my qmf.console.Console sub-class
instance's callback methods for both V1 and V2 updates.
There are some differences in the plumbing:
V1 initial updates carry properties only. V2 initial updates carry
both properties and statistics. Both are sent to the objectProps()
callback.
Subsequent updates carry only statistics, for both V1 and V2. However,
for V1 these updates are sent to the objectStats() callback, while for V2
they are sent to the objectProps() callback. (I wonder if this was
intended, or whether these were intended to go to objectStats() in V2 as
well).
Hereafter "oid" is an instance of qmf.console.ObjectId acquired by calling
the getObjectId() method of the qmf.console.Object instance that is passed
to the callbacks.
Yes, str(oid) is much prettier for V2. Having the final component be the
queue name clearly assures stability across broker restarts and unrelated
configurations. I had trouble with that belief for the hash that is the
final component of the same string for V1.
But the first 4 (dash separated) components of str(oid) don't seem useful
for my purpose of a within broker unique and restart stable identifier.
(Those first four come from oid.getFlags(), oid.getSequence(),
oid.getBrokerBank(), and oid.getAgentBank(), and turn into single digit
numbers for my test case.) I think that oid.getObject() (returns
oid.objectName, using this directly might be clearer) is more than enough
(on a V2 ObjectId instance, anyway). Please let me know if you disagree.
oid.isV2 seems to be the correct place to check to know which sort of
record I have. For now, it I only want to process one kind of record (the
useful V1 and V2 information is redundant) I'm going to use this flag.
However, it would be nice to know how to get the console to only subscribe
to the V2 queues. It's not clear that I can require the brokers to be
reconfigured as no v1, so it would be nice if there were a console option.
I'll do a bit more reading of the source code, but if anyone can shortcut
that, I'd appreciate it.
Bill
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
Thanks.
I'm snowed under. I'll try the patch in a day or so.
Bill
On Wed, Apr 3, 2013 at 12:56 PM, Ken Giusti <kg...@redhat.com> wrote:
>
>
> ----- Original Message -----
> > From: "Bill Freeman" <ke...@gmail.com>
> > To: users@qpid.apache.org
> > Sent: Wednesday, April 3, 2013 9:45:49 AM
> > Subject: Re: Questions from a novice
> >
> > On Tue, Apr 2, 2013 at 5:22 PM, Ken Giusti <kg...@redhat.com> wrote:
> >
> > > When console.py connects to a broker (via addBroker), it will attempt
> to
> > > query for the existence of the qmf.default.direct and qmf.default.topic
> > > exchanges. If that query succeeds, the console marks that broker as
> being
> > > V2 capable, and uses V2. Otherwise it falls back to V1.
> > >
> > > Take a look at the Broker::_tryToConnect() method in console.py for all
> > > the gory details.
> > >
> > >
> > And, in fact, my broker has those exchanges, and console.py finds them
> and
> > sets the Broker instance's brokerSupportsV2 attribute to True. And the
> > brokerAgent it gets has isV2 True. And it creates its v2 queues,
> > binds/subscribes them, etc.
> >
> > So I'm guessing that the rcvOjbects flag to the Session constructor only
> > gets me V1 updates, and I would have to subscribe explicitly to the
> queues
> > that I care about in order to get v2 updates. Does that sound right?
>
> No - you _should_ be getting V2 updates. But your not because of the
> aforementioned bug:
>
> https://issues.apache.org/jira/browse/QPID-4689
>
> If you apply the patch given in that above JIRA to your copy of
> console.py, you should start receiving V2 object updates.
>
> I've also found a problem with heartbeats - until I fix that you'll not
> get V2 heartbeat indications.
>
>
> In summary, once I fix that bug you should be getting async QMF v2 style
> updates.
>
>
>
> >
> > Bill
> >
>
> --
> -K
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Questions from a novice
Posted by Ken Giusti <kg...@redhat.com>.
----- Original Message -----
> From: "Bill Freeman" <ke...@gmail.com>
> To: users@qpid.apache.org
> Sent: Wednesday, April 3, 2013 9:45:49 AM
> Subject: Re: Questions from a novice
>
> On Tue, Apr 2, 2013 at 5:22 PM, Ken Giusti <kg...@redhat.com> wrote:
>
> > When console.py connects to a broker (via addBroker), it will attempt to
> > query for the existence of the qmf.default.direct and qmf.default.topic
> > exchanges. If that query succeeds, the console marks that broker as being
> > V2 capable, and uses V2. Otherwise it falls back to V1.
> >
> > Take a look at the Broker::_tryToConnect() method in console.py for all
> > the gory details.
> >
> >
> And, in fact, my broker has those exchanges, and console.py finds them and
> sets the Broker instance's brokerSupportsV2 attribute to True. And the
> brokerAgent it gets has isV2 True. And it creates its v2 queues,
> binds/subscribes them, etc.
>
> So I'm guessing that the rcvOjbects flag to the Session constructor only
> gets me V1 updates, and I would have to subscribe explicitly to the queues
> that I care about in order to get v2 updates. Does that sound right?
No - you _should_ be getting V2 updates. But your not because of the aforementioned bug:
https://issues.apache.org/jira/browse/QPID-4689
If you apply the patch given in that above JIRA to your copy of console.py, you should start receiving V2 object updates.
I've also found a problem with heartbeats - until I fix that you'll not get V2 heartbeat indications.
In summary, once I fix that bug you should be getting async QMF v2 style updates.
>
> Bill
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 03/04/13 14:45, Bill Freeman wrote:
>
> And, in fact, my broker has those exchanges,
For info I think that every qpid version from at least 0.8 has supported
QMF2 ()in the C++ broker) so these exchanges are almost certain to be
present unless you have really old brokers around (and if you do I'd
recommend you suggest to whoever owns them to upgrade)
> and console.py finds them and
> sets the Broker instance's brokerSupportsV2 attribute to True. And the
> brokerAgent it gets has isV2 True. And it creates its v2 queues,
> binds/subscribes them, etc.
>
> So I'm guessing that the rcvOjbects flag to the Session constructor only
> gets me V1 updates, and I would have to subscribe explicitly to the queues
> that I care about in order to get v2 updates. Does that sound right?
My guess is that if the initial Broker object set up has successfully
queried for the presence of QMF2 exchanges then you'd get V2 updates.
Actually I've looked at "class Session:" and the following code in
_bindingKeys suggests to me that for the case of asynchronous receipt of
broker data pushes the class is setting up subscriptions to both QMF1
and QMF2 topics
def _bindingKeys(self):
v1KeyList = []
v2KeyList = []
v1KeyList.append("schema.#")
# note well: any binding that starts with 'agent.ind.heartbeat' will be
# bound to the heartbeat queue, otherwise it will be bound to the
# unsolicited indication queue. See _decOutstanding() for the binding.
if not self.userBindings:
if self.rcvObjects and self.rcvEvents and self.rcvHeartbeats:
v1KeyList.append("console.#")
v2KeyList.append("agent.ind.data.#")
v2KeyList.append("agent.ind.event.#")
v2KeyList.append("agent.ind.heartbeat.#")
else:
# need heartbeats for V2 newAgent()/delAgent()
v2KeyList.append("agent.ind.heartbeat.#")
if self.rcvObjects:
v1KeyList.append("console.obj.#")
v2KeyList.append("agent.ind.data.#")
else:
v1KeyList.append("console.obj.*.*.org.apache.qpid.broker.agent")
if self.rcvEvents:
v1KeyList.append("console.event.#")
v2KeyList.append("agent.ind.event.#")
else:
v1KeyList.append("console.event.*.*.org.apache.qpid.broker.agent")
if self.rcvHeartbeats:
v1KeyList.append("console.heartbeat.#")
else:
# mandatory bindings
v1KeyList.append("console.obj.*.*.org.apache.qpid.broker.agent")
v1KeyList.append("console.event.*.*.org.apache.qpid.broker.agent")
v1KeyList.append("console.heartbeat.#") # no way to turn this on
later
v2KeyList.append("agent.ind.heartbeat.org_apache.qpidd.#")
return (v1KeyList, v2KeyList)
and later in bindClass() I see:
for broker in self.brokers:
if broker.isConnected():
broker.amqpSession.exchange_bind(exchange="qpid.management",
queue=broker.topicName, binding_key=v1key)
if broker.brokerSupportsV2:
# data indications should arrive on the unsolicited
indication queue
broker.amqpSession.exchange_bind(exchange="qmf.default.topic",
queue=broker.v2_topic_queue_ui, binding_key=v2key)
Which looks like it first binds from the QMF1 qpid.management topic
exchange to some queue with the v1key then if brokerSupportsV2 is set
binds from the QMF2 qmf.default.topic exchange to some other queue with
the v2key
I couldn't say for sure without looking at the code in more detail but
my guess is that data is being pushed from the broker in both QMF1 and
QMF2 format and is probably being placed on both of the queues above, in
your case the data is being read off the QMF2 queue and I'd bet that the
QMF1 queue just fills up and circulates (it's a ring queue) you could
probably check this by monitoring the depth of the queue whose name
starts "topic-", the QMF2 topic queue starts "qmfc-v2-ui-".
I'm no expert on this code, so it's just a guess on my part, I'm kind of
curious myself now :-)
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
On Tue, Apr 2, 2013 at 5:22 PM, Ken Giusti <kg...@redhat.com> wrote:
> When console.py connects to a broker (via addBroker), it will attempt to
> query for the existence of the qmf.default.direct and qmf.default.topic
> exchanges. If that query succeeds, the console marks that broker as being
> V2 capable, and uses V2. Otherwise it falls back to V1.
>
> Take a look at the Broker::_tryToConnect() method in console.py for all
> the gory details.
>
>
And, in fact, my broker has those exchanges, and console.py finds them and
sets the Broker instance's brokerSupportsV2 attribute to True. And the
brokerAgent it gets has isV2 True. And it creates its v2 queues,
binds/subscribes them, etc.
So I'm guessing that the rcvOjbects flag to the Session constructor only
gets me V1 updates, and I would have to subscribe explicitly to the queues
that I care about in order to get v2 updates. Does that sound right?
Bill
Re: Questions from a novice
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Hi Ken,
I guess the bit that you are alluding to below is:
##
## Check to see if the broker has QMFv2 exchanges configured
##
direct_result = self.amqpSession.exchange_query("qmf.default.direct")
topic_result = self.amqpSession.exchange_query("qmf.default.topic")
self.brokerSupportsV2 = not (direct_result.not_found or
topic_result.not_found)
I'm guessing that self.amqpSession.exchange_query relates to some AMQP
0.10 specific code similar to queue_declare/exchange_declare etc. I'm
not wildly familiar with those constructs as I've tried to stick to the
qpid::messaging and JMS APIs wherever I can, the stuff above appears to
be somewhat similar to things I've seen in qpid::client, though I've
generally avoided that like the plague.
I guess querying for the presence of the QMF2 exchanges in this way
avoids the conundrum of which QMF version to use to query for the
presence of the exchanges :-)
But OTOH am I correct that the constructs above relate to the pure
python implementation of AMQP 0.10? I suppose one thing to consider is
that there's now also an alternative python API/implementation based on
SWIG bindings to qpid::messaging and (I guess) the "low level" AMQP 0.10
constructs being used in console.py aren't available in the
new/alternative python API.
I guess following on from my usual broken record about cohesion and
communication it might be good if we can come out with a statement of
direction/roadmap with respect to some APIs. Given that proton/AMQP 1.0
is starting to take off and as discussed in other threads
qpid::messaging is here to stay and given some comments from Gordon that
it's likely that a pure python AMQP 1.0 is likely to be down the
priority list given the SWIG binding to qpid::messaging is it time to
start calling time on the current pure python AMQP 0.10. If not exactly
deprecate it yet then at least suggest the likely trajectory with some
putative dates.
I certainly wouldn't want to panic anyone!!!!!, this is purely my own
take, but I think without a few lines in the sand being drawn, even if
they are initially sketchy we really run the risk of confusing users as
the journey towards AMQP 1.0 progresses.
Is this worth discussing, or is everyone content with the status quo
position?
As a related aside ISTR one of my colleagues mention a memory leak with
the qpid::messaging python binding. I don't really use python in anger
myself so it's a bit third hand, is this something you are aware of? I
know that there were issues with perl that have recently been fixed, but
the comment made to me was about python. I'd definitely be interested to
know.
Frase.
On 02/04/13 22:22, Ken Giusti wrote:
> When console.py connects to a broker (via addBroker), it will attempt to query for the existence of the qmf.default.direct and qmf.default.topic exchanges. If that query succeeds, the console marks that broker as being V2 capable, and uses V2. Otherwise it falls back to V1.
>
> Take a look at the Broker::_tryToConnect() method in console.py for all the gory details.
>
> -K
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Ken Giusti <kg...@redhat.com>.
When console.py connects to a broker (via addBroker), it will attempt to query for the existence of the qmf.default.direct and qmf.default.topic exchanges. If that query succeeds, the console marks that broker as being V2 capable, and uses V2. Otherwise it falls back to V1.
Take a look at the Broker::_tryToConnect() method in console.py for all the gory details.
-K
----- Original Message -----
> From: "Fraser Adams" <fr...@blueyonder.co.uk>
> To: users@qpid.apache.org
> Cc: users@qpid.apache.org
> Sent: Tuesday, April 2, 2013 2:01:52 PM
> Subject: Re: Questions from a novice
>
> Hi Ken,
> I hadn't realised that the asynchronous python API actually had QMF2 support
> at all, I has assumed that both the original APIs were QMF1 only. I'm
> definitely learning something - though I'm not especially fond of that
> particular API, I find it a bit fiddly.
>
> It has been a long while since I did anything with that API - how do you get
> the API to choose between QMF1 or QMF2? I know at the protocol level for
> QMF2 it's a subscription to qmf.default.topic/agent.ind.* or something
> similar but I don't recall noticing how to select one protocol or the other
> via that API?
>
> Frase
>
> On 2 Apr 2013, at 14:26, Ken Giusti <kg...@redhat.com> wrote:
>
> > Hi Frase,
> >
> > What I saw surprised me: when I disabled v1 (via --mgmt-qmf1 no) and I ran
> > qpid-tool, qpid-tool didn't get any updates (no objects listed).
> >
> > As you point out, the broker should be sending unsolicited object update
> > notifications periodically. By default, each update message is sent in
> > both v1 and v2 format (from what I remember). So by turning off v1, I
> > still expected the broker to send the v2 updates.
> >
> > From what I could tell, the broker _was_ sending v2 updates - dumping
> > exchange statistics (qpid-stat -e) showed activity on the
> > qmf.default.topic. So why wasn't qpid-tool showing the objects (it uses
> > the Console callback method AFAIK).
> >
> > Turns out there's a bug in the object update handling code in console.py.
> > See the patch in https://issues.apache.org/jira/browse/QPID-4689
> >
> > From what I can tell, the console.py expects the management agent of the
> > local broker to identify itself as "broker" via the "qmf.agent" header in
> > the message. This assumption appears to have broken: I've observed the
> > current behavior as the broker is now sending it's fully qualified agent
> > name in the "qmf.agent" field instead (this would be in the form
> > <vendor>:<product>:<instance> - in the case of the broker it would be
> > "apache.org:qpidd:<xxx>" where <xxx> defaults to the mgmt uuid.
> >
> > I'm not sure when this broke, or why. But I think the "broker" convention
> > is a left over from qmfv1 support. With V2 the broker's agent name should
> > be that fully qualified triplet from above.
> >
> > The patch changes the code to recognize either format. Seems to work -
> > qpid-tool now reports all the objects if the broker is started with
> > --mgmt-qmf1 no.
> >
> > I suspect if we ever do away with v1 support, we could probably simplify
> > the console.py code quite a bit. I find it's current form a bit
> > squirrelly and loaded with special cases in order to support both qmf
> > versions.
> >
> > -K
> >
> > ----- Original Message -----
> >> From: "Fraser Adams" <fr...@blueyonder.co.uk>
> >> To: users@qpid.apache.org
> >> Cc: users@qpid.apache.org
> >> Sent: Tuesday, April 2, 2013 3:52:44 AM
> >> Subject: Re: Questions from a novice
> >>
> >>
> >>
> >> On 1 Apr 2013, at 21:31, Ken Giusti <kg...@redhat.com> wrote:
> >>
> >>> Hi Bill,
> >>>
> >>> I've just tried to restrict my broker to using only qmf v2:
> >>>
> >>> $ qpidd --auth no --mgmt-qmf1 no
> >>>
> >>> and I no longer get update notifications from my console.py app! From
> >>> the
> >>> looks of it, the console.py code is broken - the broker is generating the
> >>> qmfv2 updates, but console.py is dropping them. This is on trunk - I'll
> >>> open a JIRA.
> >> Ken, surely what you are observing is the correct behaviour? With QMF1
> >> there
> >> were two python APIs one (the most commonly used one) was synchronous and
> >> mainly used the getObjects() calls to query for objects of a given type
> >> e.g.
> >> queue, exchange, bind etc. the other API, used by I think qpid-queue-stat
> >> and I think being referred to in this thread is asynchronous. The
> >> asynchronous API relies on the broker data push and a subscription on a
> >> given topic so if you do --mgmt-qmf1 no then the broker data push to that
> >> topic, which pushes things like heartbeats and statistics updates, won't
> >> occur.
> >>
> >> Unless of course you are actually saying that disabling QMF1 data push
> >> actually stops the QMF2 data push??? I've not checked for that - it
> >> wouldn't
> >> totally surprise me as the two blocks of code in the broker are fairly
> >> intertwined and it's more likely that people would use the default
> >> behaviour
> >> with both QMF1 and QMF2 enabled.
> >>
> >> Hmm re-reading your post you do seem to be talking about perhaps something
> >> different? I am perhaps jumping to conclusions based on Bill's comments
> >> that
> >> he's using QMF1?
> >>
> >> Frase
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> >> For additional commands, e-mail: users-help@qpid.apache.org
> >
> > --
> > -K
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Hi Ken,
I hadn't realised that the asynchronous python API actually had QMF2 support at all, I has assumed that both the original APIs were QMF1 only. I'm definitely learning something - though I'm not especially fond of that particular API, I find it a bit fiddly.
It has been a long while since I did anything with that API - how do you get the API to choose between QMF1 or QMF2? I know at the protocol level for QMF2 it's a subscription to qmf.default.topic/agent.ind.* or something similar but I don't recall noticing how to select one protocol or the other via that API?
Frase
On 2 Apr 2013, at 14:26, Ken Giusti <kg...@redhat.com> wrote:
> Hi Frase,
>
> What I saw surprised me: when I disabled v1 (via --mgmt-qmf1 no) and I ran qpid-tool, qpid-tool didn't get any updates (no objects listed).
>
> As you point out, the broker should be sending unsolicited object update notifications periodically. By default, each update message is sent in both v1 and v2 format (from what I remember). So by turning off v1, I still expected the broker to send the v2 updates.
>
> From what I could tell, the broker _was_ sending v2 updates - dumping exchange statistics (qpid-stat -e) showed activity on the qmf.default.topic. So why wasn't qpid-tool showing the objects (it uses the Console callback method AFAIK).
>
> Turns out there's a bug in the object update handling code in console.py. See the patch in https://issues.apache.org/jira/browse/QPID-4689
>
> From what I can tell, the console.py expects the management agent of the local broker to identify itself as "broker" via the "qmf.agent" header in the message. This assumption appears to have broken: I've observed the current behavior as the broker is now sending it's fully qualified agent name in the "qmf.agent" field instead (this would be in the form <vendor>:<product>:<instance> - in the case of the broker it would be "apache.org:qpidd:<xxx>" where <xxx> defaults to the mgmt uuid.
>
> I'm not sure when this broke, or why. But I think the "broker" convention is a left over from qmfv1 support. With V2 the broker's agent name should be that fully qualified triplet from above.
>
> The patch changes the code to recognize either format. Seems to work - qpid-tool now reports all the objects if the broker is started with --mgmt-qmf1 no.
>
> I suspect if we ever do away with v1 support, we could probably simplify the console.py code quite a bit. I find it's current form a bit squirrelly and loaded with special cases in order to support both qmf versions.
>
> -K
>
> ----- Original Message -----
>> From: "Fraser Adams" <fr...@blueyonder.co.uk>
>> To: users@qpid.apache.org
>> Cc: users@qpid.apache.org
>> Sent: Tuesday, April 2, 2013 3:52:44 AM
>> Subject: Re: Questions from a novice
>>
>>
>>
>> On 1 Apr 2013, at 21:31, Ken Giusti <kg...@redhat.com> wrote:
>>
>>> Hi Bill,
>>>
>>> I've just tried to restrict my broker to using only qmf v2:
>>>
>>> $ qpidd --auth no --mgmt-qmf1 no
>>>
>>> and I no longer get update notifications from my console.py app! From the
>>> looks of it, the console.py code is broken - the broker is generating the
>>> qmfv2 updates, but console.py is dropping them. This is on trunk - I'll
>>> open a JIRA.
>> Ken, surely what you are observing is the correct behaviour? With QMF1 there
>> were two python APIs one (the most commonly used one) was synchronous and
>> mainly used the getObjects() calls to query for objects of a given type e.g.
>> queue, exchange, bind etc. the other API, used by I think qpid-queue-stat
>> and I think being referred to in this thread is asynchronous. The
>> asynchronous API relies on the broker data push and a subscription on a
>> given topic so if you do --mgmt-qmf1 no then the broker data push to that
>> topic, which pushes things like heartbeats and statistics updates, won't
>> occur.
>>
>> Unless of course you are actually saying that disabling QMF1 data push
>> actually stops the QMF2 data push??? I've not checked for that - it wouldn't
>> totally surprise me as the two blocks of code in the broker are fairly
>> intertwined and it's more likely that people would use the default behaviour
>> with both QMF1 and QMF2 enabled.
>>
>> Hmm re-reading your post you do seem to be talking about perhaps something
>> different? I am perhaps jumping to conclusions based on Bill's comments that
>> he's using QMF1?
>>
>> Frase
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>
> --
> -K
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Ken Giusti <kg...@redhat.com>.
Hi Frase,
What I saw surprised me: when I disabled v1 (via --mgmt-qmf1 no) and I ran qpid-tool, qpid-tool didn't get any updates (no objects listed).
As you point out, the broker should be sending unsolicited object update notifications periodically. By default, each update message is sent in both v1 and v2 format (from what I remember). So by turning off v1, I still expected the broker to send the v2 updates.
>From what I could tell, the broker _was_ sending v2 updates - dumping exchange statistics (qpid-stat -e) showed activity on the qmf.default.topic. So why wasn't qpid-tool showing the objects (it uses the Console callback method AFAIK).
Turns out there's a bug in the object update handling code in console.py. See the patch in https://issues.apache.org/jira/browse/QPID-4689
>From what I can tell, the console.py expects the management agent of the local broker to identify itself as "broker" via the "qmf.agent" header in the message. This assumption appears to have broken: I've observed the current behavior as the broker is now sending it's fully qualified agent name in the "qmf.agent" field instead (this would be in the form <vendor>:<product>:<instance> - in the case of the broker it would be "apache.org:qpidd:<xxx>" where <xxx> defaults to the mgmt uuid.
I'm not sure when this broke, or why. But I think the "broker" convention is a left over from qmfv1 support. With V2 the broker's agent name should be that fully qualified triplet from above.
The patch changes the code to recognize either format. Seems to work - qpid-tool now reports all the objects if the broker is started with --mgmt-qmf1 no.
I suspect if we ever do away with v1 support, we could probably simplify the console.py code quite a bit. I find it's current form a bit squirrelly and loaded with special cases in order to support both qmf versions.
-K
----- Original Message -----
> From: "Fraser Adams" <fr...@blueyonder.co.uk>
> To: users@qpid.apache.org
> Cc: users@qpid.apache.org
> Sent: Tuesday, April 2, 2013 3:52:44 AM
> Subject: Re: Questions from a novice
>
>
>
> On 1 Apr 2013, at 21:31, Ken Giusti <kg...@redhat.com> wrote:
>
> > Hi Bill,
> >
> > I've just tried to restrict my broker to using only qmf v2:
> >
> > $ qpidd --auth no --mgmt-qmf1 no
> >
> > and I no longer get update notifications from my console.py app! From the
> > looks of it, the console.py code is broken - the broker is generating the
> > qmfv2 updates, but console.py is dropping them. This is on trunk - I'll
> > open a JIRA.
> >
> Ken, surely what you are observing is the correct behaviour? With QMF1 there
> were two python APIs one (the most commonly used one) was synchronous and
> mainly used the getObjects() calls to query for objects of a given type e.g.
> queue, exchange, bind etc. the other API, used by I think qpid-queue-stat
> and I think being referred to in this thread is asynchronous. The
> asynchronous API relies on the broker data push and a subscription on a
> given topic so if you do --mgmt-qmf1 no then the broker data push to that
> topic, which pushes things like heartbeats and statistics updates, won't
> occur.
>
> Unless of course you are actually saying that disabling QMF1 data push
> actually stops the QMF2 data push??? I've not checked for that - it wouldn't
> totally surprise me as the two blocks of code in the broker are fairly
> intertwined and it's more likely that people would use the default behaviour
> with both QMF1 and QMF2 enabled.
>
> Hmm re-reading your post you do seem to be talking about perhaps something
> different? I am perhaps jumping to conclusions based on Bill's comments that
> he's using QMF1?
>
> Frase
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 1 Apr 2013, at 21:31, Ken Giusti <kg...@redhat.com> wrote:
> Hi Bill,
>
> I've just tried to restrict my broker to using only qmf v2:
>
> $ qpidd --auth no --mgmt-qmf1 no
>
> and I no longer get update notifications from my console.py app! From the looks of it, the console.py code is broken - the broker is generating the qmfv2 updates, but console.py is dropping them. This is on trunk - I'll open a JIRA.
>
Ken, surely what you are observing is the correct behaviour? With QMF1 there were two python APIs one (the most commonly used one) was synchronous and mainly used the getObjects() calls to query for objects of a given type e.g. queue, exchange, bind etc. the other API, used by I think qpid-queue-stat and I think being referred to in this thread is asynchronous. The asynchronous API relies on the broker data push and a subscription on a given topic so if you do --mgmt-qmf1 no then the broker data push to that topic, which pushes things like heartbeats and statistics updates, won't occur.
Unless of course you are actually saying that disabling QMF1 data push actually stops the QMF2 data push??? I've not checked for that - it wouldn't totally surprise me as the two blocks of code in the broker are fairly intertwined and it's more likely that people would use the default behaviour with both QMF1 and QMF2 enabled.
Hmm re-reading your post you do seem to be talking about perhaps something different? I am perhaps jumping to conclusions based on Bill's comments that he's using QMF1?
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Ken Giusti <kg...@redhat.com>.
----- Original Message -----
> From: "Bill Freeman" <ke...@gmail.com>
> To: users@qpid.apache.org
> Sent: Monday, April 1, 2013 7:50:16 PM
> Subject: Re: Questions from a novice
>
> Ken,
>
>
> On Mon, Apr 1, 2013 at 5:58 PM, Ken Giusti <kg...@redhat.com> wrote:
>
> > > That sounds delux, since they would probably even survive moving a broker
> > > from one server to another (with a different IP address.
> >
> > Whoa, moving a broker won't guarantee the fed-tag would be the same :(
> >
> > You see, the fed-tag is written to disk by the broker at startup, so it
> > can be preserved across restarts. But if you move the broker to a new
> > machine, and it cannot find the fed-tag it has written to disk, it will
> > create a new one. Not something you'd want. You'd have to move the
> > contents of the working directory along with it.
> >
>
>
> I presume it's written to the data directory, yes? So if the data
> directory is copied, it should keep its tag? Or is this a bad idea because
> the data directory thinks it knows IP addresses of other federation members
> (like Vulcan), even if they were originally specified by DNS name? (The
> moving is not particularly important. I'm just trying to see how much
> state I can keep if a box dies and the broker gets moved pending
> replacement.)
>
Just to be complete - yes, the generated tag is written to the data dir used by the qpidd daemon. You can control the location of this file using --data-dir command line option or config file entry. Currently, the only stuff stored in that file shouldn't depend on the original DNS name, ip address, etc. So you should be safe moving that data dir when you move the broker.
But, having said that, my own personal preference would be to configure a federation tag on a per broker instance, and assign that via the qpidd.conf file. It just seems less error prone, but this is just my gut feel. I would do this by hand-generating a unique UUID for each broker (using whatever uuid generating tools you prefer), then dole them out. But I'm a bit of a control freak :)
>
>
> >
> > You can assign a "static" fed-tag to a broker using the federation-tag
> > configuration option (either on the command line or via the qpid.config
> > file). If you deploy that way you have total control over how the fed-tag
> > is used.
> >
>
>
> This sounds like a reasonable policy for us.
>
>
>
> >
> > And to be clear, if a broker is a member of a federation, then the tag is
> > required. And it must be unique across all brokers reachable by the
> > federation. And the tag isn't modified by the brokers, so it will be
> > consistent across all brokers in the federation.
> >
>
>
> Good!
>
>
> >
> > FYI: opened a jira on that console.py issue:
> > https://issues.apache.org/jira/browse/QPID-4689
> >
>
>
> Thanks for that.
>
>
> And thanks in general.
My pleasure!
>
> Bill
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
Ken,
On Mon, Apr 1, 2013 at 5:58 PM, Ken Giusti <kg...@redhat.com> wrote:
> > That sounds delux, since they would probably even survive moving a broker
> > from one server to another (with a different IP address.
>
> Whoa, moving a broker won't guarantee the fed-tag would be the same :(
>
> You see, the fed-tag is written to disk by the broker at startup, so it
> can be preserved across restarts. But if you move the broker to a new
> machine, and it cannot find the fed-tag it has written to disk, it will
> create a new one. Not something you'd want. You'd have to move the
> contents of the working directory along with it.
>
I presume it's written to the data directory, yes? So if the data
directory is copied, it should keep its tag? Or is this a bad idea because
the data directory thinks it knows IP addresses of other federation members
(like Vulcan), even if they were originally specified by DNS name? (The
moving is not particularly important. I'm just trying to see how much
state I can keep if a box dies and the broker gets moved pending
replacement.)
>
> You can assign a "static" fed-tag to a broker using the federation-tag
> configuration option (either on the command line or via the qpid.config
> file). If you deploy that way you have total control over how the fed-tag
> is used.
>
This sounds like a reasonable policy for us.
>
> And to be clear, if a broker is a member of a federation, then the tag is
> required. And it must be unique across all brokers reachable by the
> federation. And the tag isn't modified by the brokers, so it will be
> consistent across all brokers in the federation.
>
Good!
>
> FYI: opened a jira on that console.py issue:
> https://issues.apache.org/jira/browse/QPID-4689
>
Thanks for that.
And thanks in general.
Bill
Re: Questions from a novice
Posted by Ken Giusti <kg...@redhat.com>.
> That sounds delux, since they would probably even survive moving a broker
> from one server to another (with a different IP address.
Whoa, moving a broker won't guarantee the fed-tag would be the same :(
You see, the fed-tag is written to disk by the broker at startup, so it can be preserved across restarts. But if you move the broker to a new machine, and it cannot find the fed-tag it has written to disk, it will create a new one. Not something you'd want. You'd have to move the contents of the working directory along with it.
You can assign a "static" fed-tag to a broker using the federation-tag configuration option (either on the command line or via the qpid.config file). If you deploy that way you have total control over how the fed-tag is used.
And to be clear, if a broker is a member of a federation, then the tag is required. And it must be unique across all brokers reachable by the federation. And the tag isn't modified by the brokers, so it will be consistent across all brokers in the federation.
FYI: opened a jira on that console.py issue: https://issues.apache.org/jira/browse/QPID-4689
----- Original Message -----
> From: "Bill Freeman" <ke...@gmail.com>
> To: users@qpid.apache.org
> Sent: Monday, April 1, 2013 5:21:46 PM
> Subject: Re: Questions from a novice
>
> Ken,
>
>
> On Mon, Apr 1, 2013 at 4:31 PM, Ken Giusti <kg...@redhat.com> wrote:
>
> >
> > 2) uniquely identifying objects across multiple brokers. Yes, Gordon's
> > correct, object names are usually only guaranteed to be unique within a
> > broker, not across them. Partitioning them by IP address + port could
> > work, as would the federation tag from the vhost object.
> >
> >
> > That sounds delux, since they would probably even survive moving a broker
> from one server to another (with a different IP address.
>
> But I would need it to be that:
> 1. They are required.
> 2. They are consistent no matter which broker they are viewed from.
> 3. They are forced to be unique across the federation.
>
> Is all that true?
>
> Bill
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
Ken,
On Mon, Apr 1, 2013 at 4:31 PM, Ken Giusti <kg...@redhat.com> wrote:
>
> 2) uniquely identifying objects across multiple brokers. Yes, Gordon's
> correct, object names are usually only guaranteed to be unique within a
> broker, not across them. Partitioning them by IP address + port could
> work, as would the federation tag from the vhost object.
>
>
> That sounds delux, since they would probably even survive moving a broker
from one server to another (with a different IP address.
But I would need it to be that:
1. They are required.
2. They are consistent no matter which broker they are viewed from.
3. They are forced to be unique across the federation.
Is all that true?
Bill
Re: Questions from a novice
Posted by Ken Giusti <kg...@redhat.com>.
Hi Bill,
I've just tried to restrict my broker to using only qmf v2:
$ qpidd --auth no --mgmt-qmf1 no
and I no longer get update notifications from my console.py app! From the looks of it, the console.py code is broken - the broker is generating the qmfv2 updates, but console.py is dropping them. This is on trunk - I'll open a JIRA.
Other notes:
1) re: qpidtoollib - doesn't sound like this would work for you since you need async object update notifications. Currently qpidtoollibs only support query/response and event monitoring. We probably should add async object updates, too, but right now that's not there.
2) uniquely identifying objects across multiple brokers. Yes, Gordon's correct, object names are usually only guaranteed to be unique within a broker, not across them. Partitioning them by IP address + port could work, as would the federation tag from the vhost object.
----- Original Message -----
> From: "Bill Freeman" <ke...@gmail.com>
> To: users@qpid.apache.org
> Sent: Monday, April 1, 2013 1:35:54 PM
> Subject: Re: Questions from a novice
>
> On Mon, Apr 1, 2013 at 12:21 PM, Gordon Sim <gs...@redhat.com> wrote:
>
> > On 04/01/2013 05:06 PM, Bill Freeman wrote:
> >
> >> since I have a dog and pony show coming up shortly, do you
> >> know how to get the existing qmf.console stuff to send my updates as V2
> >> (session has rcvObjects True).
> >>
> >
> > One option might be to turn v1 off at the broker... --mgmt-qmf1=false to
> > qpidd. I don't know hoe effective that will be, just a suggestion.
> >
> > Thanks. I may test with that, but for production I can't be sure that
> some department that I don't know about is using a V1 only tool.
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
Fraser
On Tue, Apr 2, 2013 at 1:48 PM, Fraser Adams
<fr...@blueyonder.co.uk>wrote:
>
> > I found this (https://cwiki.apache.org/qpid/qmfv2-project-page.html). I
> > probably didn't go here the first time around because the link to it on
> > https://cwiki.apache.org/qpid/qpid-management-framework.html says that
> it
> > is "for information about the future of QMF", so I figured that it didn't
> > apply yet.
> That's the stuff I was referring to.
>
I have been reading this (except for the "design" link, which I apparently
have to be logged in in order to see). I was tickled by the claim that one
can manage a federation through a single connection in QMF2, which seems
reasonable. I had gotten the impression earlier in this thread that maybe
this could happen, but it would take work to set up. So maybe this
requires links that the C++ agents aren't yet configuring automagically?
Or would it always require some by hand configuration, beyond that normally
needed for a federation? (Not necessarily a Fraser question, but the kind
of thing I have been prodded into thinking about.)
>
> "didn't apply yet" is slightly ironic :-/, so the QMF2 protocol stuff
> definitely applies and is worth a read, the API is slightly a sore point. I
> believe that there is a python implementation in the "extras" directory
> that implements a reasonable subset of the API and my Java stuff went
> perhaps a little OTT as I implemented the whole lot - console, agent, query
> subscriptions, everything :-)
>
> Unfortunately it hasn't really taken off, the QMF2 versions of the python
> tools such as qpid-config actually use qpidlibtools which is an alternative
> API, I'm not sure that I totally agree with all the decisions taken with
> that, it's very broker agent specific and I'm not keen on ObjectIds being
> used in a non-opaque manner. that's all by the by though, the C++ QMF2
> stuff is even odder and has its own API that doesn't seem to be documented
> anywhere, I'm not sure where that originated.
>
> Most C++ QMF2 related things I've seen seem to use the protocol directly.
> I'm personally not keen on that. Although it's based on Map messages
> there's enough boiler plate to make an API useful and given that AMQP
> supports binary and UTF8 strings and multiple different integer and
> floating point widths there's a good argument for using an API to ensure
> that types are interoperable. A good chunk of my Java QmfData class was
> about defensive coding to cope with all sorts of ClassCastException hell, I
> got a real miscellany of binary and UTF8 strings from C++ agents, so I
> needed to be careful. That's definitely something that needs to be
> considered for the future management - we must do more to consider
> interoperability across languages.
>
I think that we're stuck with APIs having a fair amount of language
specific variations. As a Python guy, probably all my integer value
normalization is handled already in the lower level qpid libraries that the
consoles uses, since in python an int is an int (at least after the
unification of long and int in the more recent python versions). There are
probably things that are easy for Java but tough for python as well, but
you mentioned the integer widths.
> The bottom line is that QMF2 is somewhat fragmented, not to mention that
> it has been C++ broker specific (though I'm working to remedy that bit).
> It's something of a shame because despite its failings I've found QMF2 to
> be pretty useful and works pretty well, but the reality is that for AMQP
> 1.0 management something will need to be put in place that has wide vendor
> buy in and that most likely won't be QMF2 as it currently stands, though
> I'm certainly keen to make sure that any transition is as painless as
> possible. At the moment QMF is the only way to manage the C++ broker and
> there are a fair number of tools out there using it, so I think that any
> migration is going to have to be a bit evolutionary. It's a shame that the
> API hasn't really taken off as that might have helped isolate clients from
> changes to the underlying management protocol. As it is the QMF2 protocol
> is exposed by several fragmented, APIs which is clearly the exact opposite
> of what might be considered useful abstraction :-(
>
Am I correct that the work itme queue idea is a feature of the API, and
thus the Console super-class implementation, rather than forced by the
protocol? I'm guessing that items arrive at the console from the broker
asynchronously, when they're ready, and the queue being drained with a zero
timeout is a within the console queue of the already arrived.
Bill
Re: Questions from a novice
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
> WebSockets in the browser is event based. You register a callback for a
> new message. You send a message either because the user clicked something,
> or because a timeout expired. So threads aren't forced by WS.
Yeah I realise that, my point was that with the AJAX/REST approach everything was event driven JavaScript side too. I'm not disagreeing at all that ultimately WebSockets are the way to go, as I say it was mostly down to a combination of lack of familiarity and something of a lack of maturity of WS. As you point out the latter is decreasing with good support in Chrome, FF, Safari and recent IE. It's definitely on my TODO list, but hasn't been a priority yet. I figured my main thrust of effort should be to push the cohesion agenda across the two qpid brokers and beyond. I think that this was actually a pretty good call given some of the conversations I've recently had with some of the other guys on AMQP management. Some of the recent work has I think given some new impetus towards getting a joined up view on management.
> I found this (https://cwiki.apache.org/qpid/qmfv2-project-page.html). I
> probably didn't go here the first time around because the link to it on
> https://cwiki.apache.org/qpid/qpid-management-framework.html says that it
> is "for information about the future of QMF", so I figured that it didn't
> apply yet.
That's the stuff I was referring to.
"didn't apply yet" is slightly ironic :-/, so the QMF2 protocol stuff definitely applies and is worth a read, the API is slightly a sore point. I believe that there is a python implementation in the "extras" directory that implements a reasonable subset of the API and my Java stuff went perhaps a little OTT as I implemented the whole lot - console, agent, query subscriptions, everything :-)
Unfortunately it hasn't really taken off, the QMF2 versions of the python tools such as qpid-config actually use qpidlibtools which is an alternative API, I'm not sure that I totally agree with all the decisions taken with that, it's very broker agent specific and I'm not keen on ObjectIds being used in a non-opaque manner. that's all by the by though, the C++ QMF2 stuff is even odder and has its own API that doesn't seem to be documented anywhere, I'm not sure where that originated.
Most C++ QMF2 related things I've seen seem to use the protocol directly. I'm personally not keen on that. Although it's based on Map messages there's enough boiler plate to make an API useful and given that AMQP supports binary and UTF8 strings and multiple different integer and floating point widths there's a good argument for using an API to ensure that types are interoperable. A good chunk of my Java QmfData class was about defensive coding to cope with all sorts of ClassCastException hell, I got a real miscellany of binary and UTF8 strings from C++ agents, so I needed to be careful. That's definitely something that needs to be considered for the future management - we must do more to consider interoperability across languages.
The bottom line is that QMF2 is somewhat fragmented, not to mention that it has been C++ broker specific (though I'm working to remedy that bit). It's something of a shame because despite its failings I've found QMF2 to be pretty useful and works pretty well, but the reality is that for AMQP 1.0 management something will need to be put in place that has wide vendor buy in and that most likely won't be QMF2 as it currently stands, though I'm certainly keen to make sure that any transition is as painless as possible. At the moment QMF is the only way to manage the C++ broker and there are a fair number of tools out there using it, so I think that any migration is going to have to be a bit evolutionary. It's a shame that the API hasn't really taken off as that might have helped isolate clients from changes to the underlying management protocol. As it is the QMF2 protocol is exposed by several fragmented, APIs which is clearly the exact opposite of what might be considered useful abstraction :-(
Do keep following this list though, I think that we really need to learn from some of the lessons of QMF and I think that one of those lessons is better communication and more open discussion.
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
On Tue, Apr 2, 2013 at 4:31 AM, Fraser Adams
<fr...@blueyonder.co.uk>wrote:
>
>
> On 1 Apr 2013, at 22:15, Bill Freeman <ke...@gmail.com> wrote:
> >
> > I realize that the final numbers come out the same. But this is to be an
> > interactive tool, used by people who are panicking because some queue
> seems
> > wedged. So if they command a change, I'd really like to show them a
> > result. Within 10 seconds seems fine. Not until the next time something
> > happens to the queue probably is not.
> >
> The data push stuff also allows receipt of heartbeats messages, TBH I
> don't use the V1 APIs much but I think that you can specify what you
> receive from the broker. If you are only specifying object updates you will
> only get notified when objects have changed. If you want a regular prod
> even if nothing has actually changed you could possibly use the heartbeat
> for this.
>
Yes. There are session constructor arguments for whether to receive
objects, receive events, and receive heartbeats. You can also specify what
you are interested in by package or by package and class, and if you
specify any some of the flags are forced on, including userbindings, which
causes the package/class filtering (which occurs in the console, so this
doesn't help with bandwidth). I am using the heartbeat to detect broker
failure, with the result that I mark data from that broker as stale (change
of color). My current presumption is that agents within a browser can't
crash independently.
...
>
> We python folks have a particularly high motivation to avoid threads.
> > Tornado's event loop model (something like Twisted, but different) is
> > pretty slick on OSes that support epoll or select.
> I've no issue with that, though JavaScript has even less thread support
> and the Rest stuff I've described works really nicely via AJAX.
>
WebSockets in the browser is event based. You register a callback for a
new message. You send a message either because the user clicked something,
or because a timeout expired. So threads aren't forced by WS.
>
> >
> > By which you mean documented in the XML? Or is there a documentation
> link
> > that I haven't found? (I've read so much stuff over the last couple of
> > months that my head is still spinning pretty faxt.)
> The XML stuff covers the management schema so is very useful, but I was
> talking about the QMF2 protocol and API documentation. I don't have the URL
> handy but if you go to the qpid home page then navigate to the wiki there's
> a link off that to QMF and from there there's links to the QMF2 project
> pages with links to the protocol and API.
>
I found this (https://cwiki.apache.org/qpid/qmfv2-project-page.html). I
probably didn't go here the first time around because the link to it on
https://cwiki.apache.org/qpid/qpid-management-framework.html says that it
is "for information about the future of QMF", so I figured that it didn't
apply yet.
>
> When I get back to my main PC I should be able to help you get started
> with a subscription to the QMF2 data and heartbeat push. As Ken suggested I
> don't think that there's any API support for this in the qpidlibtools
> python stuff, but it is mostly a subscription to
> qmf.default.topic/data.ind.* with the data coming back as Map messages,
> that might be enough for your needs.
>
> Bill
Re: Questions from a novice
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 1 Apr 2013, at 22:15, Bill Freeman <ke...@gmail.com> wrote:
>
> I realize that the final numbers come out the same. But this is to be an
> interactive tool, used by people who are panicking because some queue seems
> wedged. So if they command a change, I'd really like to show them a
> result. Within 10 seconds seems fine. Not until the next time something
> happens to the queue probably is not.
>
The data push stuff also allows receipt of heartbeats messages, TBH I don't use the V1 APIs much but I think that you can specify what you receive from the broker. If you are only specifying object updates you will only get notified when objects have changed. If you want a regular prod even if nothing has actually changed you could possibly use the heartbeat for this.
>
>
> I personally don't care whether I use object IDs or not. In fact, I'd
> rather use the queue name or exchange name, etc. That's the real problem
> for me with the V1 interface. I get two separate callbacks. One has
> properties (and I think that I only get this once per object, and I've only
> looked at queues so far), and thus has the name. The other callback has
> statistics, but no properties, so I can't use the name property to look up
> the (local cached) object for which this is a statistics update. So I'm
> using str(xxx.getObjectId()) to match them up*.
That looks about right, I think the idea is to minimise bandwidth. As you say what apps normally do is to receive the property update and use the OIDs retrieved from that as a key to match subsequent statistics pushes, if you look at qpid-queue-stat that behaves like this - certainly for older qpid versions, I've not looked recently though.
I have personally found that API to be quite fiddly when it comes to tracking changes, you end up having to be pretty careful.
>
> [* I think that some of the code in qmf.console has some dictionaries
> indexed by ObjectId instances themselves, rather than their string values,
> which I'm not sure is wise. But I'd have to go back and look again to be
> sure. ]
>
I think that's true I'm pretty certain that that's the case in qpid-queue-stat, I suspect that there's some syntactic sugar under the hood which makes indexing by the ObjectId OK.
>
> We python folks have a particularly high motivation to avoid threads.
> Tornado's event loop model (something like Twisted, but different) is
> pretty slick on OSes that support epoll or select.
I've no issue with that, though JavaScript has even less thread support and the Rest stuff I've described works really nicely via AJAX.
>
> WebSockets is pretty nifty. Comet done right. Pretty easy to use. An
> awful lot of browsers support it now. There is a JavaScript library,
> socksjs (apparently more than one: socket.io), that will use it if it's
> there and emulate it if it's not, using long poll or whatever old Comet
> technology works,
Yeah I realise that, it's mainly about familiarity - or lack thereof, it is on my list of things to do to get up to speed with WebSockets. One of the things being talked about in the qpid AMQP 1.0 discussions is having the brokers support a WebSocket transport as well as a TCP one and provide a pure JavaScript AMQP 1.0 implementation, that sounds pretty cool.
> even being able to fall back to polling with GETs. Of
> course you have to code your server to handle the GETs if you want to fall
> back that far. That's not to say that I've used it. Plain WebSockets has
> been working in my test environment: Chrome, recent Firefoxes, even the
> most recent IEs.
>
It is definitely the way of the future. I need to support IE8 so I'd need the polling fallbacks. Also as I say the getWorkItems() API call actually maps pretty well to a REST long-poll so I wasn't beating myself up too much, most of the QMF2 API is synchronous really.
>
> By which you mean documented in the XML? Or is there a documentation link
> that I haven't found? (I've read so much stuff over the last couple of
> months that my head is still spinning pretty faxt.)
The XML stuff covers the management schema so is very useful, but I was talking about the QMF2 protocol and API documentation. I don't have the URL handy but if you go to the qpid home page then navigate to the wiki there's a link off that to QMF and from there there's links to the QMF2 project pages with links to the protocol and API.
When I get back to my main PC I should be able to help you get started with a subscription to the QMF2 data and heartbeat push. As Ken suggested I don't think that there's any API support for this in the qpidlibtools python stuff, but it is mostly a subscription to qmf.default.topic/data.ind.* with the data coming back as Map messages, that might be enough for your needs.
>
> Cohesion would be great.
Oh yes, it's a bit of a soap box of mine :-)
Regards,
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
Fraser,
Thanks again.
There are a few comments interspersed below, but no reply is required.
On Mon, Apr 1, 2013 at 3:48 PM, Fraser Adams
<fr...@blueyonder.co.uk>wrote:
>
>
> On 1 Apr 2013, at 19:56, Bill Freeman <ke...@gmail.com> wrote:
>
> >
> >
> > I believe that I fall into the data push camp, even though I may not be
> > using the modern means of achieving it. I create my qmf.console.Session
> > instance with rcvObjects = True. Then, after an initial one copy of
> > everything, I'm only sent updates if anything changes, and then at most
> > once every 10 seconds. I don't want to poll. So maybe the flag works
> > (though I have other concerns about a broker wide, rather than connection
> > or even session specific setting).
> >
> That's definitely a QMF1 API that you are using then, there are actually
> two python QMF1 APIs one uses getObjects() calls and the other has a
> notification type API. The 10 second interval can be charged by a broker
> switch BTW I can't recall exactly what it is off the top of my head, but if
> you do qpidd -h it should hopefully be obvious.
>
I'm actually pretty happy with 10 seconds, but I did see where to set this
in the qpidd arguments/configuration. I don't remember it either, but I
know where to look it up. It's another one of those broker wide settings
that I'd rather not enforce on other tools connecting to these brokers (if
there are any).
And yes, I agree, moving to QMF2, and later an AMQP standard, seem
desirable. But both of those steps will be for a future deliverable for me.
>
>
> As an aside you mentioned previously about cumin "missing" updates,
> possibly that's true, but it shouldn't really matter and your application
> should probably be designed to be tolerant of this sort of thing - the
> updates of things like queueDepth, msgEnqueue, msgDequeue etc. are all
> "incremental" so there's no concept of "resetting" values a client has to
> track an initial value and deltas, it's worth pointing out though that
> timestamps are sent with each update, so if an update has been delayed in
> transit you can use the update time stamp to know the timestamp from the
> broker's perspective.
>
I realize that the final numbers come out the same. But this is to be an
interactive tool, used by people who are panicking because some queue seems
wedged. So if they command a change, I'd really like to show them a
result. Within 10 seconds seems fine. Not until the next time something
happens to the queue probably is not.
> >
> >
> > Whatever its called, I need some kind of key to disambiguate objects. I
> > need them to be consistent across restarts of the broker, and maybe even
> > across some minor reconfiguration of the broker that is outside of my
> > control (such as the deletion and/or addition of queues, exchanges, and
> > bindings). Since I will be talking to multiple brokers, I need to guard
> > against the possibility that a key could be repeated in a second broker
> > (particularly when multiple brokers are federated to duplicate efforts
> as a
> > fallback if one crashes).
> It has been a while since I looked at the guts of QMF2 ObjectIds as I say
> I tend to treat them as opaque. I think that Gordon's previous comments
> about V2 OIDs looked about right, but I'd need to have a proper look. As he
> said V1 and V2 OIDs are rather different.
>
> In V2 there's an epoch value that changes across broker restarts, I think
> that only applies to the Agent name and not the OIDs, but it has been a
> while, I'll have a proper look when I get IT back if you can bear with me.
>
> You mention being consistent across broker restarts, I'm assuming you are
> talking about persistent queues? Clearly non-persistent queues will be
> different objects. I'm not sure whether the broker treats recreated
> persistent queues as the same objects each restart, possibly but I can't be
> sure, it might not even be the right thing to do that, if you did the same
> thing with say an in memory object reference they would definitely be at
> different addresses and TBH that's usually how I view ObjectIds - as a way
> of describing an address at a point in time. I'll need to take a prober
> look to ascertain the current truth though.
>
I personally don't care whether I use object IDs or not. In fact, I'd
rather use the queue name or exchange name, etc. That's the real problem
for me with the V1 interface. I get two separate callbacks. One has
properties (and I think that I only get this once per object, and I've only
looked at queues so far), and thus has the name. The other callback has
statistics, but no properties, so I can't use the name property to look up
the (local cached) object for which this is a statistics update. So I'm
using str(xxx.getObjectId()) to match them up*.
[* I think that some of the code in qmf.console has some dictionaries
indexed by ObjectId instances themselves, rather than their string values,
which I'm not sure is wise. But I'd have to go back and look again to be
sure. ]
>
> >
> >
> > I'm much better at Python than I am at Java, and my web server (which
> > doesn't speak QMF anyway, being isolated by REDIS) is tornado, a python
> > based tool.
> Fair enough if you are more familiar with python, but I really really
> would recommend moving away from QMF1 and perhaps use the lib tool API
> Gordon mentioned earlier. As I mentioned before management is likely to be
> in for something of a reshuffle, I think it's likely that there will be
> attempts to make transitioning from QMF2 as easy as possible, but that
> probably won't hold for QMF1.
>
And I'll follow along as I can.
>
>
> > And Rest doesn't sound promising for getting updates pushed to
> > me.
> >
> So you may be right, though QMF2 has a getWorkItem() call which can be
> blocking, that is used in QMF2 in place of a pure asynchronous model as
> there was talk in the spec of avoiding threading issues. I implemented this
> in REST via a long-polling call which works pretty well, my GUI actually
> works by getting notified on heartbeats and other asynchronous events such
> as queueDeclare etc. when triggered it then does a synchronous getObjects()
> call - via REST which ultimately calls my Java getObjects(). I won't get
> into the relative pros and cons of long-polling versus WebSockets, I
> generally agree that the latter is the way to go, but I needed wide browser
> support and TBH I need to get a bit more up to speed on WebSockets, it's
> definitely on my list of things to do, but the current approach works
> pretty well, and is arguably a pretty good mapping to the underlying API
> call.
>
We python folks have a particularly high motivation to avoid threads.
Tornado's event loop model (something like Twisted, but different) is
pretty slick on OSes that support epoll or select.
WebSockets is pretty nifty. Comet done right. Pretty easy to use. An
awful lot of browsers support it now. There is a JavaScript library,
socksjs (apparently more than one: socket.io), that will use it if it's
there and emulate it if it's not, using long poll or whatever old Comet
technology works, even being able to fall back to polling with GETs. Of
course you have to code your server to handle the GETs if you want to fall
back that far. That's not to say that I've used it. Plain WebSockets has
been working in my test environment: Chrome, recent Firefoxes, even the
most recent IEs.
>
>
>
> >
> >> One last thing, there is some discussion beginning about the future of
> >> management, the eventual intention is to move away from QMF towards a
> new
> >> AMQP management API, however that is in the very early stages so at the
> >> moment QMF is the only game in town for C++ broker management. There
> are a
> >> few of us very keen to make sure that transition to the new API is as
> >> painless as possible, but as I say it's very very early days and is
> mainly
> >> said to try to ensure that you bear it in mind with your application.
> I'd
> >> very definitely avoid QMF1 though QMF2 is a better bet.
> >>
> >
> >
> > And my target brokers, which I don't get to specify, are C++ brokers.
>
> These support QMF1 and QMF2, but from a client perspective I'd really
> recommend avoiding QMF1 if you possibly can.
>
As soon as I understand how not to poll...
>
> The data push stuff is supported in both QMF1 and QMF2 formats. The QMF2
> API allows a type of query subscription, but the C++ broker doesn't
> actually support this :-( I actually managed to fake this in my Java
> implementation by doing a data subscription and filtering on the client
> side to implement the query API. That said it might have been overkill, I
> went to town implementing the full QMF2 API, but it hasn't really taken off
> and the python lib tools stuff, although it implements the QMF2 protocol
> doesn't really implement a documented API.
>
By which you mean documented in the XML? Or is there a documentation link
that I haven't found? (I've read so much stuff over the last couple of
months that my head is still spinning pretty faxt.)
>
> The lack of standardisation of QMF2 and the hitherto lack of support on
> the Java broker is one of the things driving the moves to try and create a
> properly standardised AMQP 1.0 management protocol and API.
>
> I've been working on QMF2 support for the Java broker and it's something
> of a mission of mine to try and improve the overall cohesion of AMQP
> management, that was one of the key drivers behind the GUI I put together
> and the work I'm starting on the Java broker.
>
Cohesion would be great.
>
> I'll try to give a bit more detail later in the week on some of this, some
> of it will require me to have a bit of an eyeball at some code.
>
>
>
> No rush. I'll have already done whatever I'm going to do for tomorrow's
dog and pony show.
Bill
Re: Questions from a novice
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 1 Apr 2013, at 19:56, Bill Freeman <ke...@gmail.com> wrote:
>
>
> I believe that I fall into the data push camp, even though I may not be
> using the modern means of achieving it. I create my qmf.console.Session
> instance with rcvObjects = True. Then, after an initial one copy of
> everything, I'm only sent updates if anything changes, and then at most
> once every 10 seconds. I don't want to poll. So maybe the flag works
> (though I have other concerns about a broker wide, rather than connection
> or even session specific setting).
>
That's definitely a QMF1 API that you are using then, there are actually two python QMF1 APIs one uses getObjects() calls and the other has a notification type API. The 10 second interval can be charged by a broker switch BTW I can't recall exactly what it is off the top of my head, but if you do qpidd -h it should hopefully be obvious.
As an aside you mentioned previously about cumin "missing" updates, possibly that's true, but it shouldn't really matter and your application should probably be designed to be tolerant of this sort of thing - the updates of things like queueDepth, msgEnqueue, msgDequeue etc. are all "incremental" so there's no concept of "resetting" values a client has to track an initial value and deltas, it's worth pointing out though that timestamps are sent with each update, so if an update has been delayed in transit you can use the update time stamp to know the timestamp from the broker's perspective.
>
>
> Whatever its called, I need some kind of key to disambiguate objects. I
> need them to be consistent across restarts of the broker, and maybe even
> across some minor reconfiguration of the broker that is outside of my
> control (such as the deletion and/or addition of queues, exchanges, and
> bindings). Since I will be talking to multiple brokers, I need to guard
> against the possibility that a key could be repeated in a second broker
> (particularly when multiple brokers are federated to duplicate efforts as a
> fallback if one crashes).
It has been a while since I looked at the guts of QMF2 ObjectIds as I say I tend to treat them as opaque. I think that Gordon's previous comments about V2 OIDs looked about right, but I'd need to have a proper look. As he said V1 and V2 OIDs are rather different.
In V2 there's an epoch value that changes across broker restarts, I think that only applies to the Agent name and not the OIDs, but it has been a while, I'll have a proper look when I get IT back if you can bear with me.
You mention being consistent across broker restarts, I'm assuming you are talking about persistent queues? Clearly non-persistent queues will be different objects. I'm not sure whether the broker treats recreated persistent queues as the same objects each restart, possibly but I can't be sure, it might not even be the right thing to do that, if you did the same thing with say an in memory object reference they would definitely be at different addresses and TBH that's usually how I view ObjectIds - as a way of describing an address at a point in time. I'll need to take a prober look to ascertain the current truth though.
>
>
> I'm much better at Python than I am at Java, and my web server (which
> doesn't speak QMF anyway, being isolated by REDIS) is tornado, a python
> based tool.
Fair enough if you are more familiar with python, but I really really would recommend moving away from QMF1 and perhaps use the lib tool API Gordon mentioned earlier. As I mentioned before management is likely to be in for something of a reshuffle, I think it's likely that there will be attempts to make transitioning from QMF2 as easy as possible, but that probably won't hold for QMF1.
> And Rest doesn't sound promising for getting updates pushed to
> me.
>
So you may be right, though QMF2 has a getWorkItem() call which can be blocking, that is used in QMF2 in place of a pure asynchronous model as there was talk in the spec of avoiding threading issues. I implemented this in REST via a long-polling call which works pretty well, my GUI actually works by getting notified on heartbeats and other asynchronous events such as queueDeclare etc. when triggered it then does a synchronous getObjects() call - via REST which ultimately calls my Java getObjects(). I won't get into the relative pros and cons of long-polling versus WebSockets, I generally agree that the latter is the way to go, but I needed wide browser support and TBH I need to get a bit more up to speed on WebSockets, it's definitely on my list of things to do, but the current approach works pretty well, and is arguably a pretty good mapping to the underlying API call.
>
>> One last thing, there is some discussion beginning about the future of
>> management, the eventual intention is to move away from QMF towards a new
>> AMQP management API, however that is in the very early stages so at the
>> moment QMF is the only game in town for C++ broker management. There are a
>> few of us very keen to make sure that transition to the new API is as
>> painless as possible, but as I say it's very very early days and is mainly
>> said to try to ensure that you bear it in mind with your application. I'd
>> very definitely avoid QMF1 though QMF2 is a better bet.
>>
>
>
> And my target brokers, which I don't get to specify, are C++ brokers.
These support QMF1 and QMF2, but from a client perspective I'd really recommend avoiding QMF1 if you possibly can.
The data push stuff is supported in both QMF1 and QMF2 formats. The QMF2 API allows a type of query subscription, but the C++ broker doesn't actually support this :-( I actually managed to fake this in my Java implementation by doing a data subscription and filtering on the client side to implement the query API. That said it might have been overkill, I went to town implementing the full QMF2 API, but it hasn't really taken off and the python lib tools stuff, although it implements the QMF2 protocol doesn't really implement a documented API.
The lack of standardisation of QMF2 and the hitherto lack of support on the Java broker is one of the things driving the moves to try and create a properly standardised AMQP 1.0 management protocol and API.
I've been working on QMF2 support for the Java broker and it's something of a mission of mine to try and improve the overall cohesion of AMQP management, that was one of the key drivers behind the GUI I put together and the work I'm starting on the Java broker.
I'll try to give a bit more detail later in the week on some of this, some of it will require me to have a bit of an eyeball at some code.
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
Fraser,
See below, time permitting.
On Mon, Apr 1, 2013 at 2:12 PM, Fraser Adams
<fr...@blueyonder.co.uk>wrote:
> Firstly to say that I'm currently on holiday under a qpid curfew :-) so I
> can only sneak in a quick response at the moment, I'll try to give some
> proper answers to questions posed in this thread later on in the week.
>
> On Gordon's --mgmt-qmf1=false I "thought" those broker flags only actually
> related to "push" messages sent from the broker, so that"s things like
> heartbeats and the broker "data push" that can be subscribed to, I'm pretty
> sure that the request/response stuff is driven by the address used, for
> example qmf2 uses qmf.default.direct or qmf.default.topic, for qmf1 there's
> another address qpid.management??? - I'd need to check. Anyway I think
> that's how you'd select between QMF2 and QMF1 for request response type
> things. The python APIs do this transparently under the hood.
>
I believe that I fall into the data push camp, even though I may not be
using the modern means of achieving it. I create my qmf.console.Session
instance with rcvObjects = True. Then, after an initial one copy of
everything, I'm only sent updates if anything changes, and then at most
once every 10 seconds. I don't want to poll. So maybe the flag works
(though I have other concerns about a broker wide, rather than connection
or even session specific setting).
> there's been some talk about ObjectIds, now it is true to say that V2
> ObjectIds comprise a Map that may have some meaningful names, but the qmf2
> API spec says that they are supposed to be opaque, the python lib tools
> have definitely "interpreted" them, but I'm not personally convinced that
> that's a great idea. I've certainly avoided interpreting them in anything
> I've done with QMF. To be fair it is all a bit ambiguous give that the
> names in the protocol suggest something, but as I say the QMF2 API doc
> definitely says opaque, but then again the API seems to have fallen against
> the wayside rather, which is kind of a shame.
>
Whatever its called, I need some kind of key to disambiguate objects. I
need them to be consistent across restarts of the broker, and maybe even
across some minor reconfiguration of the broker that is outside of my
control (such as the deletion and/or addition of queues, exchanges, and
bindings). Since I will be talking to multiple brokers, I need to guard
against the possibility that a key could be repeated in a second broker
(particularly when multiple brokers are federated to duplicate efforts as a
fallback if one crashes).
> One comment for Bill, the GUI that I put together has been mentioned, it's
> worth taking a look at, in particular it's not just a GUI, it's backed by a
> Java implementation of the documented QMF API, so if your Web server is
> Java based that might be more convenient to use that a python API. There is
> also a Rest API for QMF and a JavaScript API that is backed by the Rest API.
>
I'm much better at Python than I am at Java, and my web server (which
doesn't speak QMF anyway, being isolated by REDIS) is tornado, a python
based tool. And Rest doesn't sound promising for getting updates pushed to
me.
> One last thing, there is some discussion beginning about the future of
> management, the eventual intention is to move away from QMF towards a new
> AMQP management API, however that is in the very early stages so at the
> moment QMF is the only game in town for C++ broker management. There are a
> few of us very keen to make sure that transition to the new API is as
> painless as possible, but as I say it's very very early days and is mainly
> said to try to ensure that you bear it in mind with your application. I'd
> very definitely avoid QMF1 though QMF2 is a better bet.
>
And my target brokers, which I don't get to specify, are C++ brokers.
>
> I'll try to give some slightly deeper responses later in the week,
> HTH,
> Fraser
>
Have a good rest of your vacation, and thanks.
Bill
Re: Questions from a novice
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Firstly to say that I'm currently on holiday under a qpid curfew :-) so I can only sneak in a quick response at the moment, I'll try to give some proper answers to questions posed in this thread later on in the week.
On Gordon's --mgmt-qmf1=false I "thought" those broker flags only actually related to "push" messages sent from the broker, so that"s things like heartbeats and the broker "data push" that can be subscribed to, I'm pretty sure that the request/response stuff is driven by the address used, for example qmf2 uses qmf.default.direct or qmf.default.topic, for qmf1 there's another address qpid.management??? - I'd need to check. Anyway I think that's how you'd select between QMF2 and QMF1 for request response type things. The python APIs do this transparently under the hood.
there's been some talk about ObjectIds, now it is true to say that V2 ObjectIds comprise a Map that may have some meaningful names, but the qmf2 API spec says that they are supposed to be opaque, the python lib tools have definitely "interpreted" them, but I'm not personally convinced that that's a great idea. I've certainly avoided interpreting them in anything I've done with QMF. To be fair it is all a bit ambiguous give that the names in the protocol suggest something, but as I say the QMF2 API doc definitely says opaque, but then again the API seems to have fallen against the wayside rather, which is kind of a shame.
One comment for Bill, the GUI that I put together has been mentioned, it's worth taking a look at, in particular it's not just a GUI, it's backed by a Java implementation of the documented QMF API, so if your Web server is Java based that might be more convenient to use that a python API. There is also a Rest API for QMF and a JavaScript API that is backed by the Rest API.
One last thing, there is some discussion beginning about the future of management, the eventual intention is to move away from QMF towards a new AMQP management API, however that is in the very early stages so at the moment QMF is the only game in town for C++ broker management. There are a few of us very keen to make sure that transition to the new API is as painless as possible, but as I say it's very very early days and is mainly said to try to ensure that you bear it in mind with your application. I'd very definitely avoid QMF1 though QMF2 is a better bet.
I'll try to give some slightly deeper responses later in the week,
HTH,
Fraser
On 1 Apr 2013, at 18:35, Bill Freeman <ke...@gmail.com> wrote:
> On Mon, Apr 1, 2013 at 12:21 PM, Gordon Sim <gs...@redhat.com> wrote:
>
>> On 04/01/2013 05:06 PM, Bill Freeman wrote:
>>
>>> since I have a dog and pony show coming up shortly, do you
>>> know how to get the existing qmf.console stuff to send my updates as V2
>>> (session has rcvObjects True).
>>
>> One option might be to turn v1 off at the broker... --mgmt-qmf1=false to
>> qpidd. I don't know hoe effective that will be, just a suggestion.
>>
>> Thanks. I may test with that, but for production I can't be sure that
> some department that I don't know about is using a V1 only tool.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
On Mon, Apr 1, 2013 at 12:21 PM, Gordon Sim <gs...@redhat.com> wrote:
> On 04/01/2013 05:06 PM, Bill Freeman wrote:
>
>> since I have a dog and pony show coming up shortly, do you
>> know how to get the existing qmf.console stuff to send my updates as V2
>> (session has rcvObjects True).
>>
>
> One option might be to turn v1 off at the broker... --mgmt-qmf1=false to
> qpidd. I don't know hoe effective that will be, just a suggestion.
>
> Thanks. I may test with that, but for production I can't be sure that
some department that I don't know about is using a V1 only tool.
Re: Questions from a novice
Posted by Gordon Sim <gs...@redhat.com>.
On 04/01/2013 05:06 PM, Bill Freeman wrote:
> since I have a dog and pony show coming up shortly, do you
> know how to get the existing qmf.console stuff to send my updates as V2
> (session has rcvObjects True).
One option might be to turn v1 off at the broker... --mgmt-qmf1=false to
qpidd. I don't know hoe effective that will be, just a suggestion.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
Ken,
Please forgive me if I say "browser" anywhere in here. I mean "broker" but
"browser" seems to fall out of my fingers in the heat of typing.
On Mon, Apr 1, 2013 at 11:36 AM, Ken Giusti <kg...@redhat.com> wrote:
> Hi Bill (and Gordon) - sorry about not chiming in a bit earlier with this.
>
> Bill - though you've probably already seen it, the management schema is
> the definitive source for the format of all management objects made
> available via QMF: qpid/specs/management-schema.xml. It also defines the
> keys (index) used by the object ids.
>
Actually, I hadn't seen that, though I presume that the schema the console
fetches for various object types is similar.
> As Gordon correctly mentioned, there are two versions of QMF supported by
> the brokers. The original version - often referred to as QMFv1 - uses a
> very terse binary representation of those management object defined in that
> schema XML file. The v1 object id is also a weird hash-based numeric
> identifier. QMFv2 improves on this by using AMQP maps to represent these
> objects (and other components, like the object ids). Maps are a big
> improvement, as they are easier to manipulate in most languages (like
> python), and are "self identifying" - you can make sense of them without
> the need for parsing out the schema, decoding binary, etc.
>
> Having said that - you should definitely be using the v2 variant. Brokers
> currently speak both (you can control this via the command line switches to
> the qpidd broker daemon, if you want to disable v1 support and only speak
> v2, for instance). But the application can use either. Most of the tools
> speak V2, and use the qpidtoollibs library to do that.
>
> Now for object ids - the v2 object id is a map that contains a few fields,
> one of which is the _object_name. The _object_name is a string that
> uniquely identifies the object. It is composed of the package, class, and
> object's identifying keys. The keys are those properties from the object's
> schema definition that are marked as index's. For example, the Queue class
> from the management-schema.xml file:
>
>
> <class name="Queue">
> <property name="vhostRef" type="objId" references="Vhost"
> access="RC" index="y" parentRef="y"/>
> <property name="name" type="sstr" access="RC" index="y"/>
>
> <property name="durable" type="bool" access="RC"/>
> <property name="autoDelete" type="bool" access="RC"/>
> <property name="exclusive" type="bool" access="RO"/>
>
> [snip]
>
> The Queue's object Id will be a map composed of the vhostRef (which is an
> object Id itself) and the queue's name. These properties have the 'index=y'
> attribute set. [I'm unsure if vhostRef is actually used. The queue name
> definitely is].
>
I'm looking to uniquely identify objects across a federation (or possibly
even unrelated brokers), across restarts and (minor) reconfigurations.
Thus I believe I have to identify the broker, and combine that, with, say,
V2 object name. Earlier in the discussions I got the impression that there
wasn't necessarily unique across brokers, or at least no better than host
IP address and broker port number(*). Unless something as good or better
in a vhostRef, I'm probably still compositing IP, port, and V2 object
name. Any thoughts?
[* I'll ignore the possibility of identical unroutable IP addresses on
separate NATed sub-nets, or machines answering to more than one IP address.]
> Now, for this case:
>
>
> > > p qmfID.asMap()
> > > {'_object_name': '18449', '_agent_name': '0'}
> > >
> > > So these still don't seem very name like to me. I was interpreting the
> > > name of the queue as being the value of the name property. I see that
> this
> > > is arriving via Agent._handleQmfV1Message(), so I guess my broker(s)
> speak
> > > V1.
>
> You're correct - this is a map representation of a V1 style object id.
> Not very pretty. If you were talking V2, that _object_name field should
> be a string composed of the concatenation of the package, class, and index
> fields [ie. queue's name] from the class definition as per the schema.
>
> You probably want to use the qpidtoollibs as Gordon suggested. It speaks
> v2, and it's used by (most) of the command line tools for accessing the QMF
> objects. See the qpid/tools/src/py directory in the repo. You can use
> qpid-config as an example.
>
>
Can I still work in an asynchronous mode with these tools?
And separately, since I have a dog and pony show coming up shortly, do you
know how to get the existing qmf.console stuff to send my updates as V2
(session has rcvObjects True).
Thanks, Bill
Re: Questions from a novice
Posted by Ken Giusti <kg...@redhat.com>.
Hi Bill (and Gordon) - sorry about not chiming in a bit earlier with this.
Bill - though you've probably already seen it, the management schema is the definitive source for the format of all management objects made available via QMF: qpid/specs/management-schema.xml. It also defines the keys (index) used by the object ids.
As Gordon correctly mentioned, there are two versions of QMF supported by the brokers. The original version - often referred to as QMFv1 - uses a very terse binary representation of those management object defined in that schema XML file. The v1 object id is also a weird hash-based numeric identifier. QMFv2 improves on this by using AMQP maps to represent these objects (and other components, like the object ids). Maps are a big improvement, as they are easier to manipulate in most languages (like python), and are "self identifying" - you can make sense of them without the need for parsing out the schema, decoding binary, etc.
Having said that - you should definitely be using the v2 variant. Brokers currently speak both (you can control this via the command line switches to the qpidd broker daemon, if you want to disable v1 support and only speak v2, for instance). But the application can use either. Most of the tools speak V2, and use the qpidtoollibs library to do that.
Now for object ids - the v2 object id is a map that contains a few fields, one of which is the _object_name. The _object_name is a string that uniquely identifies the object. It is composed of the package, class, and object's identifying keys. The keys are those properties from the object's schema definition that are marked as index's. For example, the Queue class from the management-schema.xml file:
<class name="Queue">
<property name="vhostRef" type="objId" references="Vhost" access="RC" index="y" parentRef="y"/>
<property name="name" type="sstr" access="RC" index="y"/>
<property name="durable" type="bool" access="RC"/>
<property name="autoDelete" type="bool" access="RC"/>
<property name="exclusive" type="bool" access="RO"/>
[snip]
The Queue's object Id will be a map composed of the vhostRef (which is an object Id itself) and the queue's name. These properties have the 'index=y' attribute set. [I'm unsure if vhostRef is actually used. The queue name definitely is].
Now, for this case:
> > p qmfID.asMap()
> > {'_object_name': '18449', '_agent_name': '0'}
> >
> > So these still don't seem very name like to me. I was interpreting the
> > name of the queue as being the value of the name property. I see that this
> > is arriving via Agent._handleQmfV1Message(), so I guess my broker(s) speak
> > V1.
You're correct - this is a map representation of a V1 style object id. Not very pretty. If you were talking V2, that _object_name field should be a string composed of the concatenation of the package, class, and index fields [ie. queue's name] from the class definition as per the schema.
You probably want to use the qpidtoollibs as Gordon suggested. It speaks v2, and it's used by (most) of the command line tools for accessing the QMF objects. See the qpid/tools/src/py directory in the repo. You can use qpid-config as an example.
-K
----- Original Message -----
> From: "Gordon Sim" <gs...@redhat.com>
> To: users@qpid.apache.org
> Sent: Monday, April 1, 2013 10:32:43 AM
> Subject: Re: Questions from a novice
>
> On 04/01/2013 02:12 PM, Bill Freeman wrote:
> > On Mon, Apr 1, 2013 at 8:24 AM, Bill Freeman <ke...@gmail.com> wrote:
> >
> >>
> >>
> >>
> >> On Mon, Apr 1, 2013 at 5:03 AM, Gordon Sim <gs...@redhat.com> wrote:
> >>
> >>> On 03/29/2013 04:35 PM, Bill Freeman wrote:That console API is a bit of a
> >>> mixture of QMFv1 and QMFv2. I think there is a getObjectId() method, and
> >>> you can then call asMap() on that to get the v2 form.
> >>>
> >>
> >> Thanks. I'll have a read of the method code, call it with pdb, to see
> >> what it does for me.
> >>
> >
> > OK. That gives me:
> >
> > p qmfID.asMap()
> > {'_object_name': '18449', '_agent_name': '0'}
> >
> > So these still don't seem very name like to me. I was interpreting the
> > name of the queue as being the value of the name property. I see that this
> > is arriving via Agent._handleQmfV1Message(), so I guess my broker(s) speak
> > V1.
>
> I suspect it is merely that your use of the console API is following a
> v1 path. Unless you are using a very old broker, it will speak both v2
> and v1. I'm not sufficiently knowledgeable about that API to help but
> will try and get hold of someone who is...
>
> There is a much simpler, more basic library available for v2 available
> as well:
> https://svn.apache.org/repos/asf/qpid/trunk/qpid/tools/src/py/qpidtoollibs/broker.py.
> Depending on your needs and your existing ties to the console API that
> may be worth a look.
>
> > I'm forced to re-ask whether, in the V1 context:
> > 1. Are the name properties of queues still enforced to be unique within
> > a
> > broker?
> > 2. Are those _object_name values, '18449' still guaranteed to be stable
> > across, for example, deleting a pair of queues, restarting qpidd, and
> > re-creating the queues in the other order from which they were first
> > created?
> > 3. Is that '18449' some kind of hash or what?
> >
> > Thanks, Bill
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Gordon Sim <gs...@redhat.com>.
On 04/01/2013 04:37 PM, Bill Freeman wrote:
> On Mon, Apr 1, 2013 at 10:32 AM, Gordon Sim <gs...@redhat.com> wrote:
>> There is a much simpler, more basic library available for v2 available as
>> well: https://svn.apache.org/repos/**asf/qpid/trunk/qpid/tools/src/**
>> py/qpidtoollibs/broker.py<https://svn.apache.org/repos/asf/qpid/trunk/qpid/tools/src/py/qpidtoollibs/broker.py>.
>> Depending on your needs and your existing ties to the console API that may
>> be worth a look.
>
>
> I will, indeed, look at that. I don't suppose that there are examples
> using it, or a tutorial?
No tutorial, but it is quite basic and there are a few utilities that
use it that might be useful as examples:
https://svn.apache.org/repos/asf/qpid/trunk/qpid/tools/src/py/qpid-stat
https://svn.apache.org/repos/asf/qpid/trunk/qpid/tools/src/py/qpid-config
also
https://svn.apache.org/repos/asf/qpid/trunk/qpid/cpp/src/tests/qpidt
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
On Mon, Apr 1, 2013 at 10:32 AM, Gordon Sim <gs...@redhat.com> wrote:
> On 04/01/2013 02:12 PM, Bill Freeman wrote:
>
>> On Mon, Apr 1, 2013 at 8:24 AM, Bill Freeman <ke...@gmail.com> wrote:
>>
>>
>>>
>>>
>>> On Mon, Apr 1, 2013 at 5:03 AM, Gordon Sim <gs...@redhat.com> wrote:
>>>
>>> On 03/29/2013 04:35 PM, Bill Freeman wrote:That console API is a bit of
>>>> a
>>>> mixture of QMFv1 and QMFv2. I think there is a getObjectId() method, and
>>>> you can then call asMap() on that to get the v2 form.
>>>>
>>>>
>>> Thanks. I'll have a read of the method code, call it with pdb, to see
>>> what it does for me.
>>>
>>>
>> OK. That gives me:
>>
>> p qmfID.asMap()
>> {'_object_name': '18449', '_agent_name': '0'}
>>
>> So these still don't seem very name like to me. I was interpreting the
>> name of the queue as being the value of the name property. I see that
>> this
>> is arriving via Agent._handleQmfV1Message(), so I guess my broker(s) speak
>> V1.
>>
>
> I suspect it is merely that your use of the console API is following a v1
> path. Unless you are using a very old broker, it will speak both v2 and v1.
> I'm not sufficiently knowledgeable about that API to help but will try and
> get hold of someone who is...
>
The qmf.console.Agent instance that is calling my qmf.console.Console
sub-class's objectProps and objectStats methods does report itself (using
pdb's p as:
Agent(v2) at bank 1.0 (BrokerAgent)
These are for asynchronous updates, that, I believe, come as a result of
having initialized the session with rcvObjects and userBindings both True,
and having indicated an interest in queues in the filters. I presumed that
the v1 path was because the message received parsed that way.
>
> There is a much simpler, more basic library available for v2 available as
> well: https://svn.apache.org/repos/**asf/qpid/trunk/qpid/tools/src/**
> py/qpidtoollibs/broker.py<https://svn.apache.org/repos/asf/qpid/trunk/qpid/tools/src/py/qpidtoollibs/broker.py>.
> Depending on your needs and your existing ties to the console API that may
> be worth a look.
I will, indeed, look at that. I don't suppose that there are examples
using it, or a tutorial?
>
>
> I'm forced to re-ask whether, in the V1 context:
>> 1. Are the name properties of queues still enforced to be unique
>> within a
>> broker?
>> 2. Are those _object_name values, '18449' still guaranteed to be stable
>> across, for example, deleting a pair of queues, restarting qpidd, and
>> re-creating the queues in the other order from which they were first
>> created?
>> 3. Is that '18449' some kind of hash or what?
>>
>> I'm still curious about what the '18449' is.
Thanks, Bill
Re: Questions from a novice
Posted by Gordon Sim <gs...@redhat.com>.
On 04/01/2013 02:12 PM, Bill Freeman wrote:
> On Mon, Apr 1, 2013 at 8:24 AM, Bill Freeman <ke...@gmail.com> wrote:
>
>>
>>
>>
>> On Mon, Apr 1, 2013 at 5:03 AM, Gordon Sim <gs...@redhat.com> wrote:
>>
>>> On 03/29/2013 04:35 PM, Bill Freeman wrote:That console API is a bit of a
>>> mixture of QMFv1 and QMFv2. I think there is a getObjectId() method, and
>>> you can then call asMap() on that to get the v2 form.
>>>
>>
>> Thanks. I'll have a read of the method code, call it with pdb, to see
>> what it does for me.
>>
>
> OK. That gives me:
>
> p qmfID.asMap()
> {'_object_name': '18449', '_agent_name': '0'}
>
> So these still don't seem very name like to me. I was interpreting the
> name of the queue as being the value of the name property. I see that this
> is arriving via Agent._handleQmfV1Message(), so I guess my broker(s) speak
> V1.
I suspect it is merely that your use of the console API is following a
v1 path. Unless you are using a very old broker, it will speak both v2
and v1. I'm not sufficiently knowledgeable about that API to help but
will try and get hold of someone who is...
There is a much simpler, more basic library available for v2 available
as well:
https://svn.apache.org/repos/asf/qpid/trunk/qpid/tools/src/py/qpidtoollibs/broker.py.
Depending on your needs and your existing ties to the console API that
may be worth a look.
> I'm forced to re-ask whether, in the V1 context:
> 1. Are the name properties of queues still enforced to be unique within a
> broker?
> 2. Are those _object_name values, '18449' still guaranteed to be stable
> across, for example, deleting a pair of queues, restarting qpidd, and
> re-creating the queues in the other order from which they were first
> created?
> 3. Is that '18449' some kind of hash or what?
>
> Thanks, Bill
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
On Mon, Apr 1, 2013 at 8:24 AM, Bill Freeman <ke...@gmail.com> wrote:
>
>
>
> On Mon, Apr 1, 2013 at 5:03 AM, Gordon Sim <gs...@redhat.com> wrote:
>
>> On 03/29/2013 04:35 PM, Bill Freeman wrote:That console API is a bit of a
>> mixture of QMFv1 and QMFv2. I think there is a getObjectId() method, and
>> you can then call asMap() on that to get the v2 form.
>>
>
> Thanks. I'll have a read of the method code, call it with pdb, to see
> what it does for me.
>
OK. That gives me:
p qmfID.asMap()
{'_object_name': '18449', '_agent_name': '0'}
So these still don't seem very name like to me. I was interpreting the
name of the queue as being the value of the name property. I see that this
is arriving via Agent._handleQmfV1Message(), so I guess my broker(s) speak
V1.
I'm forced to re-ask whether, in the V1 context:
1. Are the name properties of queues still enforced to be unique within a
broker?
2. Are those _object_name values, '18449' still guaranteed to be stable
across, for example, deleting a pair of queues, restarting qpidd, and
re-creating the queues in the other order from which they were first
created?
3. Is that '18449' some kind of hash or what?
Thanks, Bill
Re: Questions from a novice
Posted by Bill Freeman <ke...@gmail.com>.
On Mon, Apr 1, 2013 at 5:03 AM, Gordon Sim <gs...@redhat.com> wrote:
> On 03/29/2013 04:35 PM, Bill Freeman wrote:That console API is a bit of a
> mixture of QMFv1 and QMFv2. I think there is a getObjectId() method, and
> you can then call asMap() on that to get the v2 form.
>
Thanks. I'll have a read of the method code, call it with pdb, to see what
it does for me.
Very interesting. Btw, just in case you haven't seen it, Fraser Adams has
> developed a graphical QMF based console as well which is soon to be added
> to the repo, but is currently available from
> https://issues.apache.org/jira/browse/QPID-3675.
>
> Thanks again. I'll enjoy taking a look at that too.
Bill