You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Fraser Adams <fr...@blueyonder.co.uk> on 2014/09/17 22:42:32 UTC
Re: Java QMF library and ObjectId
Hi Michal
Epic TL;DR response follows - I'm afraid I got on a roll ..........
You are using QMF to talk to the c++ broker (qpidd) right?
Presumably when you say "the QMF library delivers this update with
ObjectId of the form..... " you are doing this via the getObjects()
call? E.g. something like:
List<QmfConsoleData> queues =
_console.getObjects("org.apache.qpid.broker", "queue");
Is that correct?
If so the code for that is in
qpid/tools/src/java/qpid-qmf2/src/main/java/org/apache/qpid/qmf2/console/Console.java
(line 697 on trunk currently).
As far as I can recall that code simply reflects the information in the
QMF object (e.g. the queue objects in your case) that was sent by the
broker. It receives the message from JMS via:
Message response =
_responder.receive(timeout*1000);
then subsequently puts the information into a List<QmfConsoleData> via
if (AMQPMessage.isAMQPList(response))
{
List<Map> mapResults =
AMQPMessage.getList(response);
partials.ensureCapacity(partials.size() + mapResults.size());
for (Map content : mapResults)
{
partials.add(new
QmfConsoleData(content, agent));
}
}
That code caters for the fact that JMS doesn't support lists terribly well
qpid/tools/src/java/qpid-qmf2/src/main/java/org/apache/qpid/qmf2/common/AMQPMessage.java
provides some code to make it a bit easier to deal with Map and List
messages in java.util Containers.
It basically first decodes the JMS message as a List<Map> because QMF2
query responses get sent by the Broker as an AMQP List of AMQP Maps. I
then go through and create a List<QmfConsoleData> where the
QmfConsoleData is basically a wrapper around a java.util.Map
I've looked at QmfConsoleData and when that gets constructed or
initialised the ObjectId instance _object_id (which is actually owned by
the parent QmfDescribed class) gets set via
new ObjectId((Map)m.get("_object_id"));
Which is explicitly getting the property named "_object_id" from the Map
used to construct the QmfConsoleData, which is as I say a property that
has been set in the QMF2 Message sent by the broker.
So in a long, roundabout, thinking out loud nutshell I'm pretty sure
that the "particular reason why the Java implementation of QMF doesn't
generate more unique ObjectId" is because it's not the Java
Implementation of QMF that is actually generating the ObjectId in the
first place, it's simply representing exactly what was sent to it by the
Broker.
You mention that you've also been using the Python QMF implementation,
can I ask which one? There are actually two Python QMF implementations
there's one that lives in:
qpid/extras/qmf/src/py/qmf/console.py
and is used by things like qpid-queue-stats and a few other things. This
is the asynchronous API and starts something like:
class BrokerManager(Console):
The other python API lives in
qpid/qpid-trunk/qpid/tools/src/py/qpidtoollibs/broker.py
And that's the one used by the python qpid-config etc.
Looking in broker.py getAllQueues calls getAllBrokerObjects which calls
_doClassQuery which does
items = []
done = False
while not done:
for item in response.content:
items.append(item)
Which again appears pretty much to be reflecting what is being sent by
the broker.
So when you say you've used Python's QMF implementation I'm suspecting
that you aren't using getAllQueues from qpidtoollibs/broker.py. but are
in fact using qmf/console.py?
(sorry this is now really TL;DR but I tend to think out loud and the
play back is I think useful - my Maths lecturer always drummed into me
"show your working" so it's all his fault :-))
Right....... I've now just proved to my satisfaction that it is indeed
the behaviour of the Broker. What I've just done is gently hacked the
qpid-config port that I've just written in JavaScript, that's a good
"control" because it's independent of either the Java QMF or
qpidtoollibs and also uses AMQP 1.0 and proton Messenger so should rule
out any accidental bias.
What I did was hack the queueListRecurse method to look like this:
for (var i = 0; i < queues.length; i++) {
var queue = queues[i];
var queueId = oid(queue._object_id);
console.log("queueId = " + queueId);
.......
So I'm basically printing out the ObjectId that I get directly from the
QMF response. In my console I see....
<start qpidd here>
./qpid-config.js -r queues
queueId =
1815:org.apache.qpid.broker:queue:4c44dfde-5ea6-4370-d102-549237ac4ac7_#
Queue '4c44dfde-5ea6-4370-d102-549237ac4ac7_#'
bind [4c44dfde-5ea6-4370-d102-549237ac4ac7_#] => ''
<that's the oid of the temporary QMF response queue BTW. In another
window I do qpid-config add queue test>
./qpid-config.js -r queues
queueId =
1815:org.apache.qpid.broker:queue:b5250fd9-80fc-4208-94e0-8e1bbc41106d_#
Queue 'b5250fd9-80fc-4208-94e0-8e1bbc41106d_#'
bind [b5250fd9-80fc-4208-94e0-8e1bbc41106d_#] => ''
queueId = 1815:org.apache.qpid.broker:queue:test
Queue 'test'
bind [test] => ''
<So now you can see the oid of the "test" queue. In the other window I
then do qpid-config del queue test>
./qpid-config.js -r queues
queueId =
1815:org.apache.qpid.broker:queue:5daeb2a4-11a3-4c4c-a9ff-eee19947a9db_#
Queue '5daeb2a4-11a3-4c4c-a9ff-eee19947a9db_#'
bind [5daeb2a4-11a3-4c4c-a9ff-eee19947a9db_#] => ''
<as you can see, it's gone again. Now I do qpid-config add queue test again>
./qpid-config.js -r queues
queueId =
1815:org.apache.qpid.broker:queue:f7e8307e-c8ce-441c-8279-f543ad91b076_#
Queue 'f7e8307e-c8ce-441c-8279-f543ad91b076_#'
bind [f7e8307e-c8ce-441c-8279-f543ad91b076_#] => ''
queueId = 1815:org.apache.qpid.broker:queue:test
Queue 'test'
bind [test] => ''
Clearly queueId = 1815:org.apache.qpid.broker:queue:test is back and has
essentially replicated your observation, but it's demonstrably yielding
what the Broker is telling it and is not a case of the QMF library
(Java, qpidtoollibs or the JavaScript) itself generating any ObjectID.
What I *think* you are observing in the qmf/console.py library is
possibly something to do with the old QMF1 messages (IIRC the Broker can
send both formats) the old QMF1 format was a binary form and the OIDs
were IIRC quite different in structure (I'm afraid I don't know too much
about QMF1).
Now that I think about it (sorry it's taken a while until the penny
finally dropped) the QMF2 specification
(https://cwiki.apache.org/confluence/display/qpid/QMFv2+API+Proposal)
actually says something about ObjectIDs and allows them to be "predictable"
"
An object identifier uniquely addresses a data object within the domain
of its managing agent. QMF represents the object identifier as an opaque
string. An object identifier can be assigned to the object by the agent
when the object is instantiated.
Alternatively, a schema can define an object identifier by defining an
ordered list of names of data items. In this case, the object identifier
string is built by concatenating the string representations of the value
of each named data item. This approach is similar to defining index
fields within a database record.
"
It's the second paragraph that is in fact important for the purposes of
your observations "concatenating the string representations of the value
of each named data item" - does this ring any bells:
1815:org.apache.qpid.broker:queue:test
TBH I should have twigged sooner, I've actually been bitten by this in
the opposite direction. As it happens I implemented a QMF2 plugin for
the JavaBroker a while back and I actually originally implemented the
ObjectIDs by assigning UUIDs, but when I started testing it I noticed it
didn't work with the Python qpid-config, though it *did* work with my
Java qpid-config port...
I eventually figured out that my Java and JavaScript qpid-configs make
no assumptions about the ObjectID, so in order to invoke CRUD methods
(like in qpid-config add queue test) they query for the "Broker"
management object and invoke methods on that, but what qpidtoollibs does
is the following:
"
def _method(self, method, arguments,
addr="org.apache.qpid.broker:broker:amqp-broker", timeout=10):
props = {'method' : 'request',
'qmf.opcode' : '_method_request',
'x-amqp-0-10.app-id' : 'qmf2'}
correlator = str(self.next_correlator)
self.next_correlator += 1
content = {'_object_id' : {'_object_name' : addr},
'_method_name' : method,
'_arguments' : arguments}
message = self.message_class(
content, reply_to=self.reply_to, correlation_id=correlator,
properties=props, subject="broker")
self.tx.send(message)
response = self.reply_rx.fetch(timeout)
self.sess.acknowledge()
if response.properties['qmf.opcode'] == '_exception':
raise Exception("Exception from Agent: %r" %
response.content['_values'])
if response.properties['qmf.opcode'] != '_method_response':
raise Exception("bad response: %r" % response.properties)
return response.content['_arguments']
"
In other words it uses "org.apache.qpid.broker:broker:amqp-broker" which
is actually the ObjectID of the Broker Management Object, so
qpidtoollibs is choosing not to query for the Broker Management Object
to retrieve its ObjectID because it knows that the C++ Broker has
implemented ObjectIDs according to the second paragraph I quoted above.
So in precis:
1) The Java, Python qpidtoollibs and JavaScript QMF2 code are all
behaving as expected and are not in fact generating their own ObjectIDs
they are using what the Broker sent them.
2) The Broker is indeed behaving in the way that you observed, but is in
fact behaving correctly according to the QMF2 Specification.
Sorry it took so long to reach a conclusion that you probably didn't
want to hear, but hopefully the journey to the conclusion was
enlightening - and at least shows I'm not making it all up :-D
Sorry this probably doesn't help with your problem. I'm a bit torn - as
I say I originally implemented OIDs using UUIDs and changed the
behaviour to be consistent with the C++ Broker and qpidtoollibs, but
OTOH I suspect that for many people (if not the vast majority -
excepting you obviously) If you create a queue called "test" it's
probably actually quite useful to be able to refer to it by name
org.apache.qpid.broker:queue:test and if you delete it and re-add it
there's likely a reasonable argument either way as to whether you might
want it to be the *same* Object (e.g. the same object name in the same
logical namespace such as a queue and this approach gives your IDs
temporal persistence even surviving Broker restarts) the other side of
the coin is that having them unique is also not necessarily an
unreasonable assumption too - but that I'm afraid is not the way it's
interpreted by the Broker, and making them unique would itself break
qpid-config and most likely a bunch of other QMF tools, so it's probably
not going to gain much traction trying to push for a change in behaviour.
Re "It is quite important for us, since we want to keep track of also
deleted objects on our side. However, if the QMF library gives us object
with the same ObjectId (which we already keep as deleted) we have a
collision. "
I think TBH that you are simply going to have to change your algorithm.
In practice if you delete a queue named test_queue then re-add the queue
test_queue you've basically restored exactly the same QMF object as far
as the Broker is concerned, so technically you don't have a collision
you have an ObjectID that refers to the same Object and the deletion of
test_queue was merely transient behaviour.
Now if you actually *care* that the queue has been temporarily deleted
(perhaps for auditing purposes or whatever) then you have a few options
- but as far as the Broker is concerned they are the *same* Object.
If you want to identify that this has occurred you need to bear in mind
that although you say "we are using Java QMF library for receiving
asynchronously broker's objects updates " if you are using the
getObjects() stuff you are not actually working asynchronously you are
infact polling.
To deal more asynchronously you could either use QMF2 Events (which are
delivered asynchronously) and either use the info from the Event or use
that to trigger a call to getObjects() - look in
qpid/qpid-trunk/qpid/tools/src/java/qpid-qmf2-tools/src/main/java/org/apache/qpid/qmf2/tools
The ConnectionAudit.java and ConnectionLogger.java examples both
basically receive QMF Events
if (wi instanceof EventReceivedWorkItem)
.....
Then subsequently go on to do a bunch of getObjects() calls, so you
would see IIRC queueDeclare/queueDelete events that would relate to your
"test_queue"
Alternatively you could make use of the QMF2 API "Query Subscription"
mechanism. If you look in QpidQueueStats.java I've given an example of
how to use that, now I *personally* wouldn't go down that route.
Although my Java QMF2 API has implemented everything specified in
https://cwiki.apache.org/confluence/display/qpid/QMFv2+API+Proposal (as
you may have gathered I'm just a bit OCD) as it happens the Java QMF
implementation is actually the only API that ever actually got round to
implementing all of that...... what basically happened was the
realisation that the QMF2 protocol is really when all is said and done
just a bunch of Map Messages, so there's a bit of an argument whether
one should use an API or simply just use the Map Messages. My view is
that there's more of an argument for an API in strongly typed languages
like Java and C++ (most of QmfConsoleData is actually defensive code
that I discovered the hard way 'cause qpidd sends strings as AMQP byte
and I got killed by ClassCastException when I tried to cast byte[] to
String :o)). It's also woth pointing out that there's likely to be a (I
suspect vaguely slow) migration from QMF to AMQP 1.0 Management
Specification and at some point when I get my act together I'll probably
try to provide a mechanism to get info from both protocols to ease the
transition, at which point I *might* deprecate some of the more
fancy/fiddly features of the QMF2 API 'cause I suspect that nobody has
actually used them since I wrote that demo...... If you stick to
getObjects and the Event stuff I'm more likely to try to make that
polyglot than the Query Subscription stuff, which was flipping fiddly to
implement if I'm honest.
I really hope that this has been useful even if not quite what you'd
hoped for, sorry it ended up as "War and Peace", but as a result you now
probably know about as much as anyone does on QMF :->
Finally, although I don't mind answering PM'd questions I'd be grateful
if you could post any other qpid related questions to the user list -
it's mainly because there's a fighting change that if you have a Qpid
related question you won't be the only one who might like to know the
answer.
Cheers,
Frase
On 17/09/14 12:07, michal.zerola@gmail.com wrote:
> Hello Fraser,
>
> we are using Java QMF library for receiving asynchronously broker's objects updates. I have a question regarding the ObjectId - which is and object identifying the broker's object change.
>
> Imagine I create a queue test_queue on the broker, the QMF library delivers this update with ObjectId of the form (toString()):
>
> @2@org.apache.qpid.broker:queue:test_queue
>
> Now I delete the queue and I get the update identified again with the same ObjectId. So far so good. But if I create the queue again (with the same name), QMF library sends me the update again with the same ObjectId. This is different comparing to e.g. Python's QMF implementation, where I would get the new ID.
>
> It is quite important for us, since we want to keep track of also deleted objects on our side. However, if the QMF library gives us object with the same ObjectId (which we already keep as deleted) we have a collision.
>
> So questions from my side:
> - Is there a particular reason why the Java implementation of QMF doesn't generate more unique ObjectId (like Python implementation does)?
> - Would it be possible to change this behavior? We can implement it by ourselves and provide the patch to the Qpid community if you can give us brief instructions on it.
>
> Thank you in advance!
>
> Best regards,
>
> Michal
>
>
> _____________________________________
> Sent from http://qpid.2158936.n2.nabble.com
>
Re: Java QMF library and ObjectId
Posted by Michal Zerola <mi...@gmail.com>.
Hello Fraser,
thank you once again. I have verified that Python qmf/console.py is using
QMF v1 (as well as qpid-tool for example) and that is why I see different
(unique) ObjectIds than comparing to Java which uses QMF v2.
We will use some different mechanism on our side which is able to deal with
that naming.
Thanks for all the explanation regarding the Java implementation.
Cheers!
Michal
--
View this message in context: http://qpid.2158936.n2.nabble.com/Re-Java-QMF-library-and-ObjectId-tp7613784p7613940.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Java QMF library and ObjectId
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 18/09/14 10:36, Michal Zerola wrote:
> Hello Fraser and thanks for an exhaustive answer.
No probs, was that exhaustive or exhausting :-D
>
> There are still some foggy parts in this behavior for me, but let me clarify
> firstly some parts which were not explained clearly from my side:
>
> * we are connecting to the C++ Qpid broker
> * we are really using asynchronous mechanism from Java QMF API:
>
> From what you have written we might be really the only users using
> query+subscribe mechanism :) but we are quite satisfied with it.
Firstly I'm pleased that you are quite satisfied with it, quite a bit of
effort ended up going into the Query Subscription stuff for reasons I'll
mention below.
Cards on the table, I actually wrote the Java QMF2 API > 3 years ago and
it's been pretty stable, so I've not really had to do too much
tinkering, so I'm a bit rusty on some of this myself (you are now the
expert, enjoy ;->). As it happens I've never really needed to use the
Query Subscription stuff in my own applications so writing the
QpidQueueStats demo was pretty much the last time I actually used it.
There's a little bit of history, basically "back in the day" I came
across the QMF2 API that I referenced the other day and it seemed like a
good idea to implement it and as I'm slightly obsessive I ended up
implementing it all - imagine my surprise when I realised that more or
less everyone using QMF2 was doing their own thing, hence the C++
library API is totally different to my Java one and as I mentioned there
are *two* python ones etc. etc. and indeed many people simply roll their
own forsaking APIs altogether - take a look at Gordon's "Re: QPID C++ -
Dynamically Managing Broker" post from yesterday to see what I mean.
It's one of those things really, c'est la vie :-) So really there ended
up being a lot of fairly sophisticated stuff in the Java implementation
of the API that *nobody* else implemented nor is anyone ever likely too,
hence me mentioning the possibility that I might look to
simplify/rationalise/cull when I finally get round to trying to tie up
QMF with the Management spec. I'm not saying I will, I just want to give
you a little bit of fair warning just in case.
Another thing that you may not be aware of is that Query Subscriptions
to the C++ broker "cheat". In my own Agent code I actually properly
implemented the server side of the Query Subscription, but for the case
of the C++ broker Management Agent this was never implemented (as I say
nobody else ended up making use of some of this fancier stuff). However
the C++ broker *does* send periodic "_data_indication" data pushes, so I
intercept those on the client (Console) side and create "fake"
SubscriptionIndication events from them. It's a bit cheeky, but it
actually works pretty well as you've observed. If you weren't aware of
this cheekiness, then clearly it's working pretty well.
So there you have it a bit of a potted history of the Java QMF API -
take a look at the onMessage in Console.java if you are curious.
One thing that I should add though is that like the getObjects() calls
on the "synchronous" API the SubscriptionIndication resultList contains
QmfConsoleData objects that are constructed from the Map messages
received from the broker "_data_indication", so I'm afraid that, as
before, the ObjectIDs that you are seeing are exactly what has been sent
by the broker and not some fabrication made up by the Java QMF
implementation, and as I said in my mail the other day, the broker is
perfectly at liberty to use "predictable" ObjectIDs 'cause the QMF2 spec
says it can either use Unique Object IDs or the predictable ones made
from name plus namespace info.
> In
> particular, what we do is following:
>
> - create qmfPredicate describing what kind of objects are we interested in
> - create subscription based on the query using the predicate
> - register subscription to the console
> - in the onEvent() callback we receive updates and from the WorkItem we
> finally mine the List<QmfConsoleData> with that ObjectId(s) using which we
> identify what was changed/deleted/added and store it in our internal
> structures
Hopefully you've already noticed this, but the QmfConsoleData allows you
to get create/update/delete timestamps and there's also an "isDeleted"
method that lets you check if the object that has been pushed has been
deleted on the broker. If you look in QpidQueueStats I make use of that
in the block of code here....
List<QmfConsoleData> data = indication.getData();
for (QmfConsoleData record : data)
{
ObjectId id = record.getObjectId();
if (record.isDeleted())
{ // If the object was deleted by the Agent we
remove it from out Map
_objects.remove(id);
}
else
{
.....
>
> Now why do we have a problem with the 'recycled' ObjectIds. Because of
> auditing/monitoring reasons we need to keep track of also deleted objects on
> the broker (for some time).
As I said yesterday I'm afraid that you are going to have to think about
your algorithm, 'cause the broker does what it does wrt. ObjectIDs.
As mentioned above though you should be able to identify that any given
object has been deleted on the broker by using the isDeleted() call and
remove it from your local object store.
> Things are more complicated since objects can
> reference each other - e.g. binding references queue and exchange it binds
> together. However, if the references are the same for deleted and newly
> created objects (with the same name) it is quite a challenge to keep track
> of what is referencing what.
yeah I definitely know about the binding queueRef and exchangeRef
(QpidConfig uses that all over the place). "However, if the references
are the same for deleted and newly created objects " the thing is though
that when you delete a queue the bindings for that queue get deleted
too!!! So you should be able to check for the isDeleted() on the binding
objects that get pushed from the broker and remove those from your local
store too.
>
> Of course, if it is given by the broker, there is not too much we can do
> about it.
I'm afraid that's exactly the position. FWIW I don't know if you are
aware of this, but there is a pretty complete QMF GUI in the
qpid/qpid-trunk/qpid/tools/src/java/qpid-qmf2-tools/src/main/java/org/apache/qpid/qmf2/tools
area (well actually the RestAPI back end is there, the UI is in
qpid/qpid-trunk/qpid/tools/src/java/qpid-qmf2-tools/bin/qpid-web/web).
As it happens that UI (it's all JavaScript based) maintains a set of
Maps on the browser that represent a client side representation of the
broker state (there are a lot of Maps keyed by ObjectID) and that quite
happily tracks state when I add/remove queues/exchanges/bindings etc.
certainly if I add a queue then add bindings between that queue and an
exchange, then if I delete the queue and re-add it I just see the queue
object, without the bindings, 'cause they've been blatted as a result of
deleting the queue.
Basically you just have to be a little more explicit with your state
management than making an assumption that if you add an object with a
specific name multiple times the broker will treat them as unique
separate objects, as I say as far as the broker is actually concerned an
object with a given name added then deleted then added is treated as the
*same* object.
>
> Now, what still puzzles me:
>
> * from the Python I have been using qmf.console API and again it's
> asynchronous mechanism to receive object's updates. Briefly:
>
> ...Session(self.eventListener, rcvObjects=True, manageConnections=True,
> userBindings=True, rcvEvents=True)
> ...bindClass("org.apache.qpid.broker", "queue")
>
> where eventListener implements objectProps(self, broker, record) and
> objectStats(self, broker, record) callbacks. From here I am getting Object
> ID:
>
> oid = record.getObjectId()
>
> and here I can see that the ID has a different ('more unique') format: e.g.
> the same queue recreated had 0-2-1-0-15440 and later after deletion
> 0-2-1-0-1592.
That's definitely a QMF1 style ObjectID, I'm afraid that I know very
little about the python asynchronous API nor really about QMF1 - I
pretty much started out with QMF2.
>
> I can buy the argument that it uses older QMF1 protocol, but looking into
> the code of console.py I can see that it supports both QMF1 + QMF2 and if
> the QMF2 is available, the temporary request/response queues created should
> have form qmfc-v2-... - which is exactly what I can see on the broker.
I'm afraid that I can't help there, though I can recall a thread from
the Easter before last on this API, if you can find that on nabble it
might give you a few pointers.
>
> This makes me think: is my Python client communicating with QMF1 or QMF2? If
> QMF2 why do I see different ObjectId's than in Java QMF if they should be
> generated by the broker?
>
> Is there something I am missing?
What broker version are you using BTW? If you do qpidd -h you'll see
something like:
--mgmt-qmf2 yes|no (1) Enable broadcast of management
information over QMF v2
--mgmt-qmf1 yes|no (0) Enable broadcast of management
information over QMF v1
In more recent Qpid versions as you can see from the above the qmf1
broadcast is disabled by default, but in earlier versions IIRC both were
enabled. If you see something different to what I've copied above you
could have a play with these flags to see if you get results that make
some vague sense.
My guess is simply that the broker maintains both QMF1 and QMF2 style
ObjectIDs and whilst the QMF1 ones are more random the QMF2 ones are as
I've described previously.
I'm not an expert on the broker side of things, but if you look in
qpid/qpid-trunk/qpid/cpp/src/qpid/management/ManagementAgent.cpp you
will see a call to (line 358)
// Deprecated: V1 objects
ObjectId ManagementAgent::addObject(ManagementObject::shared_ptr object,
uint64_t persistId, bool persistent)
{
uint16_t sequence;
uint64_t objectNum;
sys::Mutex::ScopedLock lock(addLock);
sequence = persistent ? 0 : bootSequence;
objectNum = persistId ? persistId : nextObjectId++;
ObjectId objId(0 /*flags*/, sequence, brokerBank, objectNum);
objId.setV2Key(*object); // let object generate the v2 key
object->setObjectId(objId);
newManagementObjects.push_back(object);
QPID_LOG(debug, "Management object (V1) added: " << objId.getV2Key());
return objId;
}
Which looks to me like every time addObject() is called for the QMF1
form of objects objectNum is incremented.
whereas for V2 objects there is code:
ObjectId ManagementAgent::addObject(ManagementObject::shared_ptr object,
const string& key,
bool persistent)
{
uint16_t sequence;
sequence = persistent ? 0 : bootSequence;
ObjectId objId(0 /*flags*/, sequence, brokerBank);
if (key.empty()) {
objId.setV2Key(*object); // let object generate the key
} else {
objId.setV2Key(key);
}
object->setObjectId(objId);
{
sys::Mutex::ScopedLock lock(addLock);
newManagementObjects.push_back(object);
}
QPID_LOG(debug, "Management object added: " << objId.getV2Key());
return objId;
}
The setV2Key is in ManagementObject.cpp and looks like:
// generate the V2 key from the index fields defined
// in the schema.
void ObjectId::setV2Key(const ManagementObject& object)
{
stringstream oname;
oname << object.getPackageName() << ":" << object.getClassName() <<
":" << object.getKey();
v2Key = oname.str();
}
Which I think backs up pretty much everything I've been saying, so
basically the V1 Object IDs use an incrementing objectNum in their
construction (which will result in multiple instances of any object that
happens to have the same name be given different IDs) whereas for V2
ObjectIDs are derived from package name (e.g. org.apache.qpid.broker),
class name (e.g. queue) and "key" - which is from the "name" attribute
of the managed object, e.g. "test_queue".
>
> Thank you.
Again no problem at all, though I'm afraid that again I've clearly not
given you the answers that you might like, but I believe what I've given
you is at least comprehensive and factually correct, which will
hopefully help you to understand why you are seeing what you are seeing
a little better.
Best regards and HTH,
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Java QMF library and ObjectId
Posted by Michal Zerola <mi...@gmail.com>.
Hello Fraser and thanks for an exhaustive answer.
There are still some foggy parts in this behavior for me, but let me clarify
firstly some parts which were not explained clearly from my side:
* we are connecting to the C++ Qpid broker
* we are really using asynchronous mechanism from Java QMF API:
>From what you have written we might be really the only users using
query+subscribe mechanism :) but we are quite satisfied with it. In
particular, what we do is following:
- create qmfPredicate describing what kind of objects are we interested in
- create subscription based on the query using the predicate
- register subscription to the console
- in the onEvent() callback we receive updates and from the WorkItem we
finally mine the List<QmfConsoleData> with that ObjectId(s) using which we
identify what was changed/deleted/added and store it in our internal
structures
Now why do we have a problem with the 'recycled' ObjectIds. Because of
auditing/monitoring reasons we need to keep track of also deleted objects on
the broker (for some time). Things are more complicated since objects can
reference each other - e.g. binding references queue and exchange it binds
together. However, if the references are the same for deleted and newly
created objects (with the same name) it is quite a challenge to keep track
of what is referencing what.
Of course, if it is given by the broker, there is not too much we can do
about it.
Now, what still puzzles me:
* from the Python I have been using qmf.console API and again it's
asynchronous mechanism to receive object's updates. Briefly:
...Session(self.eventListener, rcvObjects=True, manageConnections=True,
userBindings=True, rcvEvents=True)
...bindClass("org.apache.qpid.broker", "queue")
where eventListener implements objectProps(self, broker, record) and
objectStats(self, broker, record) callbacks. From here I am getting Object
ID:
oid = record.getObjectId()
and here I can see that the ID has a different ('more unique') format: e.g.
the same queue recreated had 0-2-1-0-15440 and later after deletion
0-2-1-0-1592.
I can buy the argument that it uses older QMF1 protocol, but looking into
the code of console.py I can see that it supports both QMF1 + QMF2 and if
the QMF2 is available, the temporary request/response queues created should
have form qmfc-v2-... - which is exactly what I can see on the broker.
This makes me think: is my Python client communicating with QMF1 or QMF2? If
QMF2 why do I see different ObjectId's than in Java QMF if they should be
generated by the broker?
Is there something I am missing?
Thank you.
Cheers,
Michal
--
View this message in context: http://qpid.2158936.n2.nabble.com/Re-Java-QMF-library-and-ObjectId-tp7613784p7613803.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org