You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Justin Ross <jr...@apache.org> on 2014/04/11 00:20:04 UTC
Comments on the AMQP management spec
Hi, everyone. I've been using the draft management spec to develop some
command line tools[1], and I've got some questions and comments. These
comments are based on working draft 8 of the spec[2].
Thanks,
Justin
[1] https://github.com/ssorj/bailey (Watch out! It's a work in progress.)
[2]
https://www.oasis-open.org/committees/document.php?document_id=52425&wg_abbrev=amqp
---
# Use of AMQP application-properties
Given that values in AMQP 1.0 application properties cannot be lists or
maps (yikes--I didn't know that), I feel it's inadvisable to use
application properties to define requests. We'd be better off using
properties in the message body. Otherwise, we'll have to define standard
string encodings in every instance where we want to use a non-scalar type.
# Application properties and studlyCaps
The spec calls for studlyCaps for standard property names, whereas an
earlier draft had hyphenated names. FWIW, I prefer the latter, since it’s
in line with the core AMQP spec.
# QUERY and pagination
There doesn’t appear to be a way to query for the total count without
fetching all the values. Without that, you can’t build a data efficient
page-navigation UI that offers links to each page (or just the last page).
Requests using offset and count should perhaps instead use offset and
“limit”. That’s more familiar to those who have used SQL, and it correctly
signals that the number returned may be less than the number requested. In
my view, the request should use “limit”, and the response should use
“count”.
Also, if we’re doing paging we need to do sorting as well. Otherwise, A
client UI cannot build a sortable paginated table without pulling down
everything.
# Request-response
There’s a note in section 4 that multiple response messages may be produced
for a single request. How should a client determine whether the response
is complete?
# GET-MGMT-NODES
The text says the response message must contain “a list of addresses of
other Management Nodes”. Perhaps this should more explicitly prohibit
listing the currently-in-communication management node (or drop the word
"other").
# [DE]REGISTER
What is the intended use of this? That is, what new behavior do you get
when something is registered?
# Ping
I’d like to see a standard PING or ECHO operation on $management. I can
simulate it by using an arbitrary operation, but that seems less than
straightforward for implementors.
# Clerical stuff
3.3.3.1: Awkward phrasing: “so if any of the changes cannot be applied, the
entire operation should not be applied and to multiple values changed this
MUST result in a failure response”
3.4.1: Typo: “this operation supports pagination <though> which a request”
3.4.1: Awkward phrasing: “A result set of size N <can be considered to
containing elements>”
# Odds and ends
- 2.3: Should this reference the AMQP definition of a “node”?
- 3.1: It would be nice to mention that locales is a comma-separated list.
The RFC mentions it, but you have to dig down a bit.
- 3.3.1.2: What is a good example of a generic value made specific in an
UPDATE response?
- 3.3.3: Is the lack of attribute append a problem? Right now, there is
only replacing list values, and dueling actors could mean lost state.
Re: [amqp-comment] Re: Comments on the AMQP management spec
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
For humour RE:
The mythical AMQP toaster example...
As it happens.......
My TV is already (sort of) AMQP enabled, I've been using it (it has a
reasonably decent Safari browser with WebSocket support) to play with my
JavaScript implementation :-D
Yes, I do need to get out more .....
F.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Comments on the AMQP management spec
Posted by Rob Godfrey <ro...@gmail.com>.
OK - so Fraser beat me to it in terms of responding, and said many of the
things I was going to say :-)
On 11 April 2014 10:35, Fraser Adams <fr...@blueyonder.co.uk> wrote:
> Hi Justin & Rob
> I've replied to this on the TC comment list, but for completeness and
> transparency here's what I said:
>
> RE:
>
>
> # Application properties and studlyCaps
>
> The spec calls for studlyCaps for standard property names, whereas an
> earlier draft had hyphenated names. FWIW, I prefer the latter, since it's
> in line with the core AMQP spec.
>
>
> hyphens are an interesting case as I've found to my cost . You might have
> seen my post to the qpid user list back in February entitled "A write up of
> some AMQP 1.0 Experiments" during those experiments I discovered some
> interesting behaviours with hyphenated names and message selectors:
>
> In precis whilst hyphens are indeed legal AMQP 1.0 the *are not* legal
> wrt. JMS message selectors, indeed they are not *actually* legal JMS
> application property names (I only realised that myself when I was trying
> to figure out the selector thing). It turns out that JMS properties have to
> have names that are valid Java identifiers.
>
> Now technically that might be a moot point from the perspective of an AMQP
> Specification - after all if it's legal AMQP.... but there's no point
> making things hard for Java clients and it also doesn't seem great to have
> one AMQP Specification specifying a form of properties that would be
> illegal somewhere else:
> https://svn.apache.org/repos/asf/qpid/trunk/qpid/specs/
> apache-filters.xml#section-legacy-amqp
> Explaining the reason for the choice might be worthwhile though.
>
>
>
Yep - that's one of the primary reasons for using camel case rather than
hyphens (hyphens all make bad for identifiers in code as well).
> On a related note previous working drafts of AMQP 1.0 Management had
> specified using lists as message bodies in quite a few places, which is
> perfectly legal AMQP (though as you have noted lists and maps as property
> values is not) but because lists are a pain for JMS I argued that was
> *probably* a bad idea. I think that you do more python things than Java
> don't you? For python, C++ and pretty much everything else lists are fine,
> but for Java aaaarrrgggg!!
>
> Re: # Ping
> That's an interesting one, something like that would be useful, but I
> personally don't think a simple ping to a $management node scales, so that
> would be a very restrictive solution in a large topology of containers or
> where there are lots of clients. I included the following in an email to
> Rob but it was quite a long exchange and it may have been a bit TL;DR :-)
> but here goes.....
> One other thing that I keep forgetting to mention; in QMF2 Agents
> broadcast heartbeat messages periodically to a topic so clients can
> identify if Agents have gone away. I guess that this is perhaps more useful
> in QMF2 where Agents aren't *necessarily* co-located with the Broker, but
> it has got me thinking. In AMQP 1.0 Management I guess that the $management
> node is likely to be pretty coupled to the Container's lifecycle such that
> a client could probably infer that the $management node is gone based on
> interception of a connection failure (e.g. via something like a Connection
> Exception Listener) but a couple of use cases fall out of that sort of
> scope:
> 1) What about Management nodes other than $management? It's possible that
> the Container and $management might be available, but some other Management
> node a user is interested in might possibly die asynchronously for whatever
> reason (a simple use case might be one client DEREGISTERing a Management
> Node that another client happens to be using).
> 2) What about Message-oriented (as opposed to Connection-oriented) APIs
> such as Messenger, with those the fact of Connection closing wouldn't
> necessarily register until the next time a request was made.
>
> Clearly a Management client could figure this out by some sort of polling
> - and that might be better than needing a topic to broadcast a heartbeat
> (though it doesn't scale as well if there are lots of clients) but if
> polling is the expected way to accomplish this then I'd say that the
> Management Specification needs to specify a standard Management node
> operation to achieve this (perhaps STATUS ?). For someone designing a
> decoupled client (say a WebApp) being able to determine the status of
> Management nodes and provide feedback to the user is pretty useful.
>
>
It's something to consider - I'm not sure that status is anything different
than trying to do a READ of "self" though. We could be more explicit in
saying that the READ operation must always be permitted on "self"
Some more comments on Justin's original e-mail below...
> Regards,
> Frase
>
>
> On 11/04/14 00:07, Rob Godfrey wrote:
>
>> Hi Justin,
>>
>> can you send these comments to the public TC comment list (see [1]) so
>> they
>> can be incorporated in the discussions there on any updates to the spec
>> (in
>> your case - since you are employed by an organisation who is a member of
>> OASIS and have voting members on the TC - this may not be strictly
>> required... however it seems like good form :-) ).
>>
>> I'll try to respond on the technical points (later) in the morning.
>>
>> -- Rob
>>
>> [1] https://www.oasis-open.org/committees/comments/index.php?
>> wg_abbrev=amqp
>>
>>
>> On 11 April 2014 00:20, Justin Ross <jr...@apache.org> wrote:
>>
>> Hi, everyone. I've been using the draft management spec to develop some
>>> command line tools[1], and I've got some questions and comments. These
>>> comments are based on working draft 8 of the spec[2].
>>>
>>> Thanks,
>>> Justin
>>>
>>> [1] https://github.com/ssorj/bailey (Watch out! It's a work in
>>> progress.)
>>> [2]
>>>
>>> https://www.oasis-open.org/committees/document.php?
>>> document_id=52425&wg_abbrev=amqp
>>>
>>> ---
>>>
>>> # Use of AMQP application-properties
>>>
>>> Given that values in AMQP 1.0 application properties cannot be lists or
>>> maps (yikes--I didn't know that), I feel it's inadvisable to use
>>> application properties to define requests. We'd be better off using
>>> properties in the message body. Otherwise, we'll have to define standard
>>> string encodings in every instance where we want to use a non-scalar
>>> type.
>>>
>>
The application properties are used only to provide the highest level
information about the operation. If complex data is needed it can be
passed in the body. Burying all the data in the body makes it harder for
management nodes that are actually managing a distributed network to
effectively route actions based on the entity/operation they are applying
to.
>
>>> # Application properties and studlyCaps
>>>
>>> The spec calls for studlyCaps for standard property names, whereas an
>>> earlier draft had hyphenated names. FWIW, I prefer the latter, since
>>> it's
>>> in line with the core AMQP spec.
>>>
>>> # QUERY and pagination
>>>
>>> There doesn't appear to be a way to query for the total count without
>>> fetching all the values. Without that, you can't build a data efficient
>>> page-navigation UI that offers links to each page (or just the last
>>> page).
>>>
>>> Requests using offset and count should perhaps instead use offset and
>>> "limit". That's more familiar to those who have used SQL, and it
>>> correctly
>>> signals that the number returned may be less than the number requested.
>>> In
>>> my view, the request should use "limit", and the response should use
>>> "count".
>>>
>>>
The pattern we've generally used is to echo back the parameters sent in...
I'm not personally to fussed either way, but I'm not sure it is a huge
barrier for use. The total issue is interesting - I'd be OK optionally
supplying a total, but it may be that for some implementations it is not
possible / efficient to count all the records that might possibly be
retrieved... in which case it might be better to have "Page 2 of lots"
style ...
> Also, if we're doing paging we need to do sorting as well. Otherwise, A
>>> client UI cannot build a sortable paginated table without pulling down
>>> everything.
>>>
>>
That's just moving the burden onto the management node to first collate and
then sort the data for every request. For management nodes which are
returning large data sets I think there's a good probability that the node
in question is actually collating data from a larger network, in which case
large data transfers still have to take place... moreover we'd then need to
define a mechanism for describing sorting on different fields and how to
order complex types, etc.
>
>>> # Request-response
>>>
>>> There's a note in section 4 that multiple response messages may be
>>> produced
>>> for a single request. How should a client determine whether the response
>>> is complete?
>>>
>>>
Hmmm... that seems odd - I think multiple responses should be the preserve
of subscription style patterns (not yet defined).
> # GET-MGMT-NODES
>>>
>>> The text says the response message must contain "a list of addresses of
>>> other Management Nodes". Perhaps this should more explicitly prohibit
>>> listing the currently-in-communication management node (or drop the word
>>> "other").
>>>
>>>
Agreed
> # [DE]REGISTER
>>>
>>> What is the intended use of this? That is, what new behavior do you get
>>> when something is registered?
>>>
>>>
It allows management nodes to be registered into the network and thus
discovered. The mythical AMQP toaster example... the toaster joins the
network and tells the intermediary it connected to "hey! there's a new
management node at /amqptoaster/$management" ... Now anyone connecting to
the intermediary can discover there is a toaster there ready to be managed
(as can anyone who connected to another part of the network that was aware
of the management node of the intermediary, etc).
> # Ping
>>>
>>> I'd like to see a standard PING or ECHO operation on $management. I can
>>> simulate it by using an arbitrary operation, but that seems less than
>>> straightforward for implementors.
>>>
>>> # Clerical stuff
>>>
>>> 3.3.3.1: Awkward phrasing: "so if any of the changes cannot be applied,
>>> the
>>> entire operation should not be applied and to multiple values changed
>>> this
>>> MUST result in a failure response"
>>> 3.4.1: Typo: "this operation supports pagination <though> which a
>>> request"
>>> 3.4.1: Awkward phrasing: "A result set of size N <can be considered to
>>> containing elements>"
>>>
>>
Thx - we'll address those.
>
>>> # Odds and ends
>>>
>>> - 2.3: Should this reference the AMQP definition of a "node"?
>>>
>>
Yes it probably should
> - 3.1: It would be nice to mention that locales is a comma-separated
>>> list.
>>
>> The RFC mentions it, but you have to dig down a bit.
>>>
>>
Agreed - we should add explanatory text / an example
> - 3.3.1.2: What is a good example of a generic value made specific in an
>>> UPDATE response?
>>>
>>
3.3.1.2 is for CREATE where the example is you ask for an entity of type
org.amqp.Queue and the object you get back is an org.apache.qpid.QpidQueue
For update you may get things that differ from the request in the sense
that maybe you decided to add alerting to the queue but didn't specify a
frequency limit, so you get a frequency limit defaulted... I'm not sure on
whether there is a case where you get a more specific value - but I
wouldn't rule it out ... maybe updating the type?
> - 3.3.3: Is the lack of attribute append a problem? Right now, there is
>>> only replacing list values, and dueling actors could mean lost state.
>>>
>>
I don't think append per se is a problem, but the more general problem is
that the spec provides no way of ensuring that the object was in the
expected state when the updates were applied. One would need to be able
to add some sort of pre-condition to verify that the object had not changed
from the state the updater believed it to be in...
-- Rob
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Comments on the AMQP management spec
Posted by Gordon Sim <gs...@redhat.com>.
On 04/11/2014 11:20 AM, Rob Godfrey wrote:
> if we went with JSON we'd also have to define a canonical encoding
> of all AMQP types into JSON which we've looked at before and doesn't end up
> pretty...
Why? I think the benefit of using JSON directly is that you avoid
needing to do this.
If you go with the AMQP typesystem then you either need a whole host of
comparison/equivalence rules - i.e. essentially a canonical mapping into
a much simpler typesystem - or you have to have schemas be precise about
the exact AMQP type that some variable is (i.e. uint16 not anything else).
> And it would make things harder for JMS users, etc...
I'm not convinced it would. Even if exposed simply as JSON I don't think
it would be a huge barrier and of course it could be masked behind
ObjectMessage or whatever as with the AMQP type system if desired.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Comments on the AMQP management spec
Posted by Rob Godfrey <ro...@gmail.com>.
On 11 April 2014 11:48, Fraser Adams <fr...@blueyonder.co.uk> wrote:
> On 11/04/14 09:57, Gordon Sim wrote:
>
>>
>> In the spirit of keeping things as simple as possible, how about using
>> JSON in the body? It is already in widespread use for modelling state for
>> all sorts of things (including messaging brokers).
>>
>> I've got some sympathy with that, though TBH I quite like the AMQP Maps
> and Lists and those map pretty well to JSON, but with the added benefit of
> a type system for stored primitives, so I'm not totally convinced.
>
> If I'm really honest (and I've said this before so it won't be any
> surprise) the best thing that could be done to simplify things from a Java
> perspective it to use ObjectMessage to encode java.util.List and
> java.util.Map. Being able to use standard containers and manipulation
> thereof makes most of the pain go away, so I'm not convinced that the extra
> encoding/decoding step into a JSON string is the right thing and feels more
> of a cop-out/workaround
>
> Another thing (and this might be moot in the context of the Management
> Spec) is that it's not totally unreasonable to want to put binary data into
> a Map - one of the things I experimented with a while back was the
> possibility of using QMF method invocation as a "standardised" (I wish)
> AMQP based remote method invocation framework. It actually had everything I
> needed including a documented specification, mechanisms for schema
> distribution - the works. I still think it was quite neat and would have
> saved a ton of wheel reinvention. Anyway I used that to retrieve large
> binary payloads as part of an experiment - you could of course achieve this
> plenty of other ways, but as I say not having a mechanism for binary
> contents is perhaps an unnecessary constraint on users?
>
> Ironically I've got a feeling using JSON actually *complicates* things for
> many users. As you probably know I'm working on "productising" JavaScript
> bindings to Proton Messenger this works in a very similar way to the Python
> bindings where I can do something like:
> message.setAddress(address);
> message.setSubject(subject);
> message.body = ['Rod', 'Jane', 'Freddy', {cat: true, donkey: 'hee
> haw'}];
> messenger.put(message);
>
> So I can transparently serialise native JavaScript Arrays to AMQP list and
> Object to AMQP map.
>
> I'm quite looking forward to getting it finished so that I can mess around
> with AMQP 1.0 Management in the most enjoyable way possible. I'm not too
> keen on an extra JSON encoding/parsing step :-) though I could (and would)
> clearly hide it in my put implementation.
>
> My personal take on this is that using JSON might be nice from a client
> perspective but that the best way to achieve that might be to add a JSON
> encode/decode capability to the C++/Java tool sets and use AMQP maps and
> lists on the wire so languages that can handle these things nicely aren't
> penalised.
>
>
>
Yeah - if we went with JSON we'd also have to define a canonical encoding
of all AMQP types into JSON which we've looked at before and doesn't end up
pretty... And it would make things harder for JMS users, etc... I think
the way to go is for client libraries to have JSON -> AMQP Map translation
which hopefully we can then turn into a canonical AMQP <-> JSON mapping
which we can push through as a standard
> FWIW I suggested using Maps in the Management spec. because it was still
> in draft so figured it might be easier to stop that nobbling Java than to
> get Java to behave more sensibly :-D
>
>
> While I'm on a roll...
> Rob, could you do me a favour and make sure that the Management spec is
> *absolutely clear and explicit* (referring to the relevant section in the
> AMQP 1.0 spec would be good) that strings in management nodes and
> manageable entities should be AMQP 1.0 UTF8 strings. I realise that it's
> implicit in some of the wording, but I'm in no hurry to allow the
> possibility that management nodes (or clients) might be sending C style
> binary strings. I've been on the receiving end of too many
> ClassCastExceptions of byte[] values in maps when I was reasonably
> expecting String. I'd really like to be able to say that an application is
> off spec if it sends binary strings and be able to clearly point to a
> reference to say so #interoperability :-)
>
Yeah - we can do that... we should probably use a different font face or
something when we talk about AMQP types "string" rather than using it as a
common word
-- Rob
>
> Cheers,
> Frase
>
>
>
>
>
>
Re: Comments on the AMQP management spec
Posted by Gordon Sim <gs...@redhat.com>.
On 04/11/2014 06:36 PM, Rafael Schloming wrote:
> JSON makes exactly the same sort of
> vacuous promise. By magically limiting everyone to use only strings and
> numbers, then structured data exchange suddenly becomes trivial!
Be as dismissive as you like. The fact remains that *many* systems today
manage perfectly well with JSON, with all its imperfections.
No, it doesn't make data exchange trivial. But then I don't believe the
AMQP type system does either.
> There really isn't nearly as much room for this sort of thing as you're
> suggesting. If you allow for the automatic type conversions between numeric
> types that most languages do automatically, then the number of different
> choices you have for "intended types" is actually quite constrained.
I have first hand - though possibly vacuous - experience of bugs with
these sorts of conversions and comparisons.
[...]
> I don't think this is a valid comparison. There is exactly one way to "work
> around" the issue in the AMQP case, your implementation simply needs to do
> pretty much the same standard type conversions that most languages do
> automatically. In the case of dates/times with JSON there are a hundred
> different and non interoperable ways to "work around" the problem.
We will have to agree to disagree.
[...]
> So how would you propose dealing with things like creation/expiration dates
> and message ttls using JSON?
I would probably lean towards a string format for creation/expiration
dates (i.e. absolute times) and a numeric values representing
milliseconds for ttls and relative times. Ultimately though it would be
about reaching a consensus with as wide a range of implementers as
possible, in the context of some more specific, concrete schema alignments.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Comments on the AMQP management spec
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Fri, Apr 11, 2014 at 12:54 PM, Gordon Sim <gs...@redhat.com> wrote:
> On 04/11/2014 05:06 PM, Rafael Schloming wrote:
>
>> I don't think an encoding system actually has any impact at all on type
>> conversion/comparison issues.
>>
>
> I think it does. An encoding system that can encode 8 as many different
> types increases the chance that a given system gets it in a type it wasn't
> expecting.
My point is that types are fundamental to a given problem domain. If your
application needs to represent timestamps, then you're going to figure out
a way to do that whether or not your encoding system permits you do to it
in a standard way, and that is ultimately the source of
conversion/comparison issues. It's like the original XML myth that if
everything uses XML, data interchange is magically easy somehow. Anyone
who's seen the number of different ways people represent addresses in XML
knows that this is utterly false. JSON makes exactly the same sort of
vacuous promise. By magically limiting everyone to use only strings and
numbers, then structured data exchange suddenly becomes trivial!
>
>
> That's really a function of the language at a
>> given endpoint. The best you can do with a type system is provide a means
>> for the author of a given piece of content to fully express the intended
>> types in a standard way.
>>
>
> Lets say I want to define the number of recognised priority levels for a
> queue, the valid values for which will be 0 to 100 inclusive. What should
> the type of that be? Lets say I choose to define it as a uint8.
>
> Is it then illegal to send a legal value encoded as e.g. an int8 or a
> uint16?
>
That's really up to whoever is defining the application. The AMQP type
system is really just about encoding the data, it intentionally doesn't
mandate how you interpret/convert/cast/coerce it in your application. As a
general principal for building robust systems I would recommend following
Postel's law here[1]. "Be conservative in what you send, and be liberal in
what you accept." So I'd say you should send the value as uint8, but accept
any integral value in the appropriate range.
[1] http://en.wikipedia.org/wiki/Robustness_principle
>
> I would argue not. I would argue that the real 'intended type' is an
> integer value between 0 and 100 and the exact encoding in which it is sent
> isn't that important. However when the encoding presents lots of options
> its more likely that two parts of a system didn't pick the same option and
> further that one did not anticipate the choice of the other.
>
There really isn't nearly as much room for this sort of thing as you're
suggesting. If you allow for the automatic type conversions between numeric
types that most languages do automatically, then the number of different
choices you have for "intended types" is actually quite constrained.
>
> Can one work around this? Of course! One can also work around the issues
> with dates/times in JSON. No system will be completely free from annoyances
> or problems.
>
I don't think this is a valid comparison. There is exactly one way to "work
around" the issue in the AMQP case, your implementation simply needs to do
pretty much the same standard type conversions that most languages do
automatically. In the case of dates/times with JSON there are a hundred
different and non interoperable ways to "work around" the problem.
>
> [...]
>
> I tend to think JSON is very popular exactly because it is a reaction to
>> certain overcomplicated technologies, but as often happens in reactionary
>> situations like that it really does go a bit to far in the overly
>> simplistic direction.
>>
>
> I'm sure there are cases for which it is overly simple. However I
> personally don't think managing brokers and similar things is one of those,
> and I think using a simpler type system and encoding would make it more
> accessible and easier to adopt.
So how would you propose dealing with things like creation/expiration dates
and message ttls using JSON?
--Rafael
Re: Comments on the AMQP management spec
Posted by Gordon Sim <gs...@redhat.com>.
On 04/11/2014 05:06 PM, Rafael Schloming wrote:
> I don't think an encoding system actually has any impact at all on type
> conversion/comparison issues.
I think it does. An encoding system that can encode 8 as many different
types increases the chance that a given system gets it in a type it
wasn't expecting.
> That's really a function of the language at a
> given endpoint. The best you can do with a type system is provide a means
> for the author of a given piece of content to fully express the intended
> types in a standard way.
Lets say I want to define the number of recognised priority levels for a
queue, the valid values for which will be 0 to 100 inclusive. What
should the type of that be? Lets say I choose to define it as a uint8.
Is it then illegal to send a legal value encoded as e.g. an int8 or a
uint16?
I would argue not. I would argue that the real 'intended type' is an
integer value between 0 and 100 and the exact encoding in which it is
sent isn't that important. However when the encoding presents lots of
options its more likely that two parts of a system didn't pick the same
option and further that one did not anticipate the choice of the other.
Can one work around this? Of course! One can also work around the issues
with dates/times in JSON. No system will be completely free from
annoyances or problems.
[...]
> I tend to think JSON is very popular exactly because it is a reaction to
> certain overcomplicated technologies, but as often happens in reactionary
> situations like that it really does go a bit to far in the overly
> simplistic direction.
I'm sure there are cases for which it is overly simple. However I
personally don't think managing brokers and similar things is one of
those, and I think using a simpler type system and encoding would make
it more accessible and easier to adopt.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Comments on the AMQP management spec
Posted by Alan Conway <ac...@redhat.com>.
On Fri, 2014-04-11 at 12:06 -0400, Rafael Schloming wrote:
[snip]
>
> I tend to think JSON is very popular exactly because it is a reaction to
> certain overcomplicated technologies, but as often happens in reactionary
> situations like that it really does go a bit to far in the overly
> simplistic direction. I think with things like BSON/UBJSON/etc, and even
> with AMQP's type system itself you see the pendulum swinging back hopefully
> towards a reasonable middle ground.
Pendulums move fastest at the middle, they spend most of their time at
the extremes.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Comments on the AMQP management spec
Posted by Rafael Schloming <rh...@alum.mit.edu>.
<rant>
The problem with JSON is that it has achieved "ease and simplicity" by
arguably over constraining the types of values it can represent. There is a
bit of a false promise in the whole "look every language knows about
numbers and strings, so every language can deal with JSON" premise of the
whole thing. That's great and all, but almost no real world applications
can function purely on numbers and strings and so if your application needs
say timestamps and/or binary data, then you suddenly your standard encoding
is less of a standard and more just a style of using strings that has to be
extended to be able to truly represent values in pretty much any real world
domain. Just look at some of the fun people have trying to stuff timestamps
into JSON[1], or dealing with binary data[2]:
[1] http://markembling.info/2011/07/json-date-time
[2]
http://stackoverflow.com/questions/1443158/binary-data-in-json-string-something-better-than-base64
These sorts of problems have already motivated things like BSON and UBJSON
both of which add a significant number of types and I think also add the
ability to extend the type system and in fact use the same type/length/data
encoding format that AMQP does. Despite their motivation by JSON they
actually resemble the AMQP type system far more than they resemble JSON.
</rant>
(As an aside, don't let the XML rant tags mislead you, I'm no fan of XML
either.)
Further possibly less ranty comments inline...
On Fri, Apr 11, 2014 at 11:34 AM, Gordon Sim <gs...@redhat.com> wrote:
> On 04/11/2014 02:58 PM, Fraser Adams wrote:
>
>> On 11/04/14 13:25, Gordon Sim wrote:
>>
>>> On 04/11/2014 12:31 PM, Fraser Adams wrote:
>>>
>>>> ultimately language bindings are going to have to have ways of
>>>> dealing with that anyway
>>>>
>>>
>>> True, but there is a difference between the library having the ability
>>> to encode/decode the type system internally and exposing that in a
>>> suitable manner to the applications.
>>>
>>
>> That was exactly the point I was making earlier "exposing that in a
>> suitable manner to the applications." in the JavaScript example I
>> illustrated earlier it's currently *exactly* in a manner suitable to the
>> application and I guess python is largely the same it's only really
>> strongly typed languages like C++ and Java where it gets more excitings.
>>
>> It wouldn't totally /kill/ me to JSON.stringify:
>> message.body = JSON.stringify(['Rod', 'Jane', 'Freddy', {cat: true,
>> donkey: 'hee haw'}]);
>>
>
> I don't think you would need to do that. For example, you could allow the
> application to leave the body as a map and e.g. set a content-type to
> text/json and then have the library encode it under the covers.
>
> The point I was making is that a client library that supports AMQP, may
> still required the body of messages to be encoded/decoded by the
> application. That may not be about the language it is written in, but the
> API exposed.
>
> (This for example was how the c++ qpid::messaging API was designed to work
> originally; the API had to be extended to support AMQP 1.0 encoded
> messages).
>
> Using such a library requires you to have access to codecs for whatever
> encoding, or to create one yourself. That is where the advantage of
> something simple comes in.
>
> [...]
>
> I do generally have a lot of time for JSON, but I guess I just don't
>> really see what it buys here,
>>
>
> I think it generally makes things more accessible by virtue of being a
> simpler encoding, and minimises the scope for type conversion and
> comparison issues.
I don't think an encoding system actually has any impact at all on type
conversion/comparison issues. That's really a function of the language at a
given endpoint. The best you can do with a type system is provide a means
for the author of a given piece of content to fully express the intended
types in a standard way. If you happen to be decoding into a language that
can't easily express all the different types then so be it, but that
doesn't mean there shouldn't be a standard way of expressing what are
ultimately all very commonly used and well understood types.
Look at trying to compare date/time values in JSON vs AMQP. If the language
you've decoded your values into doesn't have a standard representation of
date/time values then you're equally screwed either way. If it does have a
standard representation then at least in the AMQP case you don't have any
issues.
>
>
> ultimately C++ and Java clients will
>> likely have application code using Maps and Lists so will need a
>> mechanism to serialise into JSON - so ultimately it's just being used as
>> a serialised wire-level format at and that fact would/could probably be
>> opaque to users of the Management Spec. but why is that better than AMQP
>> Maps and Lists.
>>
>
> It's better - in my opinion of course - because its simpler and far more
> widely used.
>
>
> I'd tend to agree about the limited type system, but as
>> I said in my earlier post specifying management node/entity strings as
>> AMQP 1.0 String and numbers as AMQP 1.0 double would be enough to
>> constrain the type system to something universal and each to convert to
>> JSON if needed somewhere else.
>>
>
> In a modified version of your original code snippet:
>
> message.body = ['Rod', 10000001, {cat: true, donkey: 8}];
>
> how would you indicate that this is a management related message and that
> therefore the numbers should always be encoded as a double? (and what about
> the boolean?)
>
> If you have a way to indicate the encoding to use, that same approach
> could be used to encode as JSON or anything else.
I think the real question here is can you get away without a standard means
of representing things like timestamps and/or ttls.
>
>
> One bit of value is that I don't have to see JSON.stringify() upsetting
>> my nice Arrays and Objects ;-) to be fair in my case it's all "mentally"
>> JSON anyway so it really is just about the wire format.
>>
>
> As above, I don't think you need to do that.
>
>
> I guess that one possible advantage of JSON is that having a simple
>> string message body *may* be more compact than AMQP list/map though
>> that's probably a marginal benefit, if at all.
>>
>
> I certainly don't care about that aspect.
>
>
> I'm still concerned about what would happen if I happened to want to
>> create a Management Node or Manageable Entity where I happened to want
>> to send it binary data, it might be an edge case and I've not got a good
>> use case off the top of my head, using AMQP maps/lists I'd have that
>> option, but not with JSON (well base64 bleughhh!!).
>>
>> It's not something I'm going to die in a ditch over, as I say I'm
>> generally quite fond of JSON (as it happens the QMF2 REST API actually
>> serialises the QMF Management Objects to JSON so it looks almost exactly
>> like you are suggesting in my HTTP responses) but in that case there
>> were fewer options - I don't need to do that when I can talk AMQP
>> directly it seems a shame pushing things all into a string if you don't
>> have to.
>>
>
> I see JSON as a simpler encoding rather than a hack that pushes everything
> around as a string. So to me, its a shame to use a more complex encoding
> when you don't need it.
>
>
> To be fair though lists are a right pain in Java, but I think that the
>> right answer is to put an place a /nice/ way of encoding/decoding lists
>> (ObjectMessage :-)) rather than duck the issue and push strings around.
>>
>
> (It's a pain in JMS, not Java per se, a different API might have no
> issue). JSON isn't a way of ducking that specific issue. To me its a
> generalisation of the notion of making things simple (i.e. avoiding making
> this more complicated than they need to be) for as wide a set of usages as
> possible.
I tend to think JSON is very popular exactly because it is a reaction to
certain overcomplicated technologies, but as often happens in reactionary
situations like that it really does go a bit to far in the overly
simplistic direction. I think with things like BSON/UBJSON/etc, and even
with AMQP's type system itself you see the pendulum swinging back hopefully
towards a reasonable middle ground.
--Rafael
Type systems - was Re: Comments on the AMQP management spec
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 11/04/14 16:34, Gordon Sim wrote:
> In a modified version of your original code snippet:
>
> message.body = ['Rod', 10000001, {cat: true, donkey: 8}];
>
> how would you indicate that this is a management related message and
> that therefore the numbers should always be encoded as a double? (and
> what about the boolean?)
That's actually a really interesting question and one that has caused me
a lot of navel gazing as to how to best encode from JavaScript into the
AMQP type system (going the other way is relatively easy - well longs
have some interesting edge cases).
I was as bit torn, for being *explicit* about types I started off
thinking about creating a raft of wrapper/boxing classes such as say
proton.Data.Float, proton.Data.UnsignedInteger etc. etc. ad infinitum
:-) though I'm not thinking that I'll use strings with a "literal" form
instead e.g. {"Rod": "4.0f"} {"Jack": "5ul"} and so on and if I want the
string 4.0f to quote it {"Rod": "'4.0f'"} that seems a reasonable
approach for typed numbers in a loosely typed language and feels perhaps
more natural and idiomatic than say {"Rod": new proton.Data.Float(4.0)}
though clearly the latter is as explicit as it gets, but is looking
closer to Java than JavaScript at that point.
I'd be interested in the thoughts of the group on this - I'm open to offers.
It's possibly more "exciting" when considering what to do by default for
numbers. The "easy and obvious" thing to do would be to take JavaScript
numbers and encode them as double, what I'm currently doing looks like
this (still work in progress) because "easy and obvious" didn't
necessarily translate in my mind to "most useful"
} else if (Data.isNumber(obj)) {
/**
* Encoding JavaScript numbers is surprisingly complex and has
several
* gotchas. The code here tries to do what the author believes
is the
* most intuitive encoding of the native JavaScript Number. It
first
* tries to identify if the number is an integer or floating
point type
* by checking if the number modulo 1 is zero (i.e. if it has a
remainder
* then it's a floating point type, which is encoded here as a
double).
* If the number is an integer type a test is made to check if
it is a
* 32 bit Int value. N.B. JavaScript automagically coerces floating
* point numbers with a zero Fractional Part into an exact
integer so
* numbers like 1.0, 100.0 etc. will be encoded as int or long
here,
* which is unlikely to be what is wanted. There's no easy way
around this
* the two main options are to add a very small fractional
number or to
* represent the number in a String literal e.g. "1.0f",
"1.0d", "1l"
*/
if (obj % 1 === 0) {
console.log(obj + " is Integer Type " + (obj|0));
if (obj === (obj|0)) { // the |0 coerces to a 32 bit value.
// 32 bit integer - encode as an Integer.
console.log(obj + " is Int ");
this['putInteger'](obj);
} else { // Longer than 32 bit - encode as a Long.
console.log(obj + " is Long");
this['putLong'](obj);
}
} else { // Floating point type - encode as a Double
console.log(obj + " is Float Type");
this['putDouble'](obj);
}
}
In other words I check if the number is an "integer like number" by
checking its remainder, if it is integer like I coerce it into a 32 bit
value and if that is the same as the original then it is a real integer
otherwise I encode as a long. Floating point types I encode as doubles.
I think that's mostly what would be "natural" in most peoples minds,
however JavaScript screws with that a bit, so to take your example
message.body = ['Rod', 10000001, {cat: true, donkey: 8}];
Those would be encoded as Integers, but unfortunately
message.body = ['Rod', 10000001.0, {cat: true, donkey: 8.0}];
Would also be encoded as Integers because JavaScript implicitly converts
things that can be exactly represented as integers into integers
internally. So something like this
message.body = ['Rod', "10000001.0", {cat: true, donkey: "8.0"}];
Would be needed to specify integer numbers as doubles if you see what I
mean.
I'm quite torn on all of this, which is why I definitely agreed with
your comment that type systems can complicate things rather :-)
Again I'd be interested in what the community thinks would be the most
useful approach. The JavaScript bindings are coming on quite well in the
branch (currently working on Binary data support). I've still got some
tidying, adding tests and sorting out some wrinkles like the stuff
above, but it's starting to look pretty decent.
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Comments on the AMQP management spec
Posted by Gordon Sim <gs...@redhat.com>.
On 04/11/2014 02:58 PM, Fraser Adams wrote:
> On 11/04/14 13:25, Gordon Sim wrote:
>> On 04/11/2014 12:31 PM, Fraser Adams wrote:
>>> ultimately language bindings are going to have to have ways of
>>> dealing with that anyway
>>
>> True, but there is a difference between the library having the ability
>> to encode/decode the type system internally and exposing that in a
>> suitable manner to the applications.
>
> That was exactly the point I was making earlier "exposing that in a
> suitable manner to the applications." in the JavaScript example I
> illustrated earlier it's currently *exactly* in a manner suitable to the
> application and I guess python is largely the same it's only really
> strongly typed languages like C++ and Java where it gets more excitings.
>
> It wouldn't totally /kill/ me to JSON.stringify:
> message.body = JSON.stringify(['Rod', 'Jane', 'Freddy', {cat: true,
> donkey: 'hee haw'}]);
I don't think you would need to do that. For example, you could allow
the application to leave the body as a map and e.g. set a content-type
to text/json and then have the library encode it under the covers.
The point I was making is that a client library that supports AMQP, may
still required the body of messages to be encoded/decoded by the
application. That may not be about the language it is written in, but
the API exposed.
(This for example was how the c++ qpid::messaging API was designed to
work originally; the API had to be extended to support AMQP 1.0 encoded
messages).
Using such a library requires you to have access to codecs for whatever
encoding, or to create one yourself. That is where the advantage of
something simple comes in.
[...]
> I do generally have a lot of time for JSON, but I guess I just don't
> really see what it buys here,
I think it generally makes things more accessible by virtue of being a
simpler encoding, and minimises the scope for type conversion and
comparison issues.
> ultimately C++ and Java clients will
> likely have application code using Maps and Lists so will need a
> mechanism to serialise into JSON - so ultimately it's just being used as
> a serialised wire-level format at and that fact would/could probably be
> opaque to users of the Management Spec. but why is that better than AMQP
> Maps and Lists.
It's better - in my opinion of course - because its simpler and far more
widely used.
> I'd tend to agree about the limited type system, but as
> I said in my earlier post specifying management node/entity strings as
> AMQP 1.0 String and numbers as AMQP 1.0 double would be enough to
> constrain the type system to something universal and each to convert to
> JSON if needed somewhere else.
In a modified version of your original code snippet:
message.body = ['Rod', 10000001, {cat: true, donkey: 8}];
how would you indicate that this is a management related message and
that therefore the numbers should always be encoded as a double? (and
what about the boolean?)
If you have a way to indicate the encoding to use, that same approach
could be used to encode as JSON or anything else.
> One bit of value is that I don't have to see JSON.stringify() upsetting
> my nice Arrays and Objects ;-) to be fair in my case it's all "mentally"
> JSON anyway so it really is just about the wire format.
As above, I don't think you need to do that.
> I guess that one possible advantage of JSON is that having a simple
> string message body *may* be more compact than AMQP list/map though
> that's probably a marginal benefit, if at all.
I certainly don't care about that aspect.
> I'm still concerned about what would happen if I happened to want to
> create a Management Node or Manageable Entity where I happened to want
> to send it binary data, it might be an edge case and I've not got a good
> use case off the top of my head, using AMQP maps/lists I'd have that
> option, but not with JSON (well base64 bleughhh!!).
>
> It's not something I'm going to die in a ditch over, as I say I'm
> generally quite fond of JSON (as it happens the QMF2 REST API actually
> serialises the QMF Management Objects to JSON so it looks almost exactly
> like you are suggesting in my HTTP responses) but in that case there
> were fewer options - I don't need to do that when I can talk AMQP
> directly it seems a shame pushing things all into a string if you don't
> have to.
I see JSON as a simpler encoding rather than a hack that pushes
everything around as a string. So to me, its a shame to use a more
complex encoding when you don't need it.
> To be fair though lists are a right pain in Java, but I think that the
> right answer is to put an place a /nice/ way of encoding/decoding lists
> (ObjectMessage :-)) rather than duck the issue and push strings around.
(It's a pain in JMS, not Java per se, a different API might have no
issue). JSON isn't a way of ducking that specific issue. To me its a
generalisation of the notion of making things simple (i.e. avoiding
making this more complicated than they need to be) for as wide a set of
usages as possible.
> It's probably a bit moot though I guess
More than likely :-)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Comments on the AMQP management spec
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 11/04/14 13:25, Gordon Sim wrote:
> On 04/11/2014 12:31 PM, Fraser Adams wrote:
>> ultimately language bindings are going to have to have ways of
>> dealing with that anyway
>
> True, but there is a difference between the library having the ability
> to encode/decode the type system internally and exposing that in a
> suitable manner to the applications.
That was exactly the point I was making earlier "exposing that in a
suitable manner to the applications." in the JavaScript example I
illustrated earlier it's currently *exactly* in a manner suitable to the
application and I guess python is largely the same it's only really
strongly typed languages like C++ and Java where it gets more excitings.
It wouldn't totally /kill/ me to JSON.stringify:
message.body = JSON.stringify(['Rod', 'Jane', 'Freddy', {cat: true,
donkey: 'hee haw'}]);
but TBH I think that it's C++ and Java that are the cause of the problem
:-) so why force ugly code onto pretty languages like Python and JavaScript.
>
> [...]
>> "For the purposes of management you really only need strings and
>> numbers" might be reasonable, but you don't necessarily need JSON to
>> enforce this, perhaps the Management Spec being explicit about UTF
>> strings and double (I guess) for numbers and you can then map pretty
>> easily to/from JSON from a management perspective.
>
> I certainly agree you don't *need* JSON. However its a well understood
> and reasonably well supported scheme that is already widely used for
> managing resources.
I do generally have a lot of time for JSON, but I guess I just don't
really see what it buys here, ultimately C++ and Java clients will
likely have application code using Maps and Lists so will need a
mechanism to serialise into JSON - so ultimately it's just being used as
a serialised wire-level format at and that fact would/could probably be
opaque to users of the Management Spec. but why is that better than AMQP
Maps and Lists. I'd tend to agree about the limited type system, but as
I said in my earlier post specifying management node/entity strings as
AMQP 1.0 String and numbers as AMQP 1.0 double would be enough to
constrain the type system to something universal and each to convert to
JSON if needed somewhere else.
>
> A restricted subset of the AMQP type system would be better than
> nothing (lest described types start appearing in 'standard' management
> schemas!), but what's the value of that over just using something that
> already exists and is simpler (yet sufficient)?
>
One bit of value is that I don't have to see JSON.stringify() upsetting
my nice Arrays and Objects ;-) to be fair in my case it's all "mentally"
JSON anyway so it really is just about the wire format.
I guess that one possible advantage of JSON is that having a simple
string message body *may* be more compact than AMQP list/map though
that's probably a marginal benefit, if at all.
I'm still concerned about what would happen if I happened to want to
create a Management Node or Manageable Entity where I happened to want
to send it binary data, it might be an edge case and I've not got a good
use case off the top of my head, using AMQP maps/lists I'd have that
option, but not with JSON (well base64 bleughhh!!).
It's not something I'm going to die in a ditch over, as I say I'm
generally quite fond of JSON (as it happens the QMF2 REST API actually
serialises the QMF Management Objects to JSON so it looks almost exactly
like you are suggesting in my HTTP responses) but in that case there
were fewer options - I don't need to do that when I can talk AMQP
directly it seems a shame pushing things all into a string if you don't
have to.
To be fair though lists are a right pain in Java, but I think that the
right answer is to put an place a /nice/ way of encoding/decoding lists
(ObjectMessage :-)) rather than duck the issue and push strings around.
It's probably a bit moot though I guess that this conversation should go
much wider than this list as it's really an OASIS discussion and I guess
that adds a whole world of different vendors into the mix :-) trying to
herd the cats into agreeing to JSON would probably push the Management
Spec back about two years ;-D
Frase
Re: Comments on the AMQP management spec
Posted by Gordon Sim <gs...@redhat.com>.
On 04/11/2014 12:31 PM, Fraser Adams wrote:
> ultimately language bindings are going to have to have ways of
> dealing with that anyway
True, but there is a difference between the library having the ability
to encode/decode the type system internally and exposing that in a
suitable manner to the applications.
[...]
> "For the purposes of management you really only need strings and
> numbers" might be reasonable, but you don't necessarily need JSON to
> enforce this, perhaps the Management Spec being explicit about UTF
> strings and double (I guess) for numbers and you can then map pretty
> easily to/from JSON from a management perspective.
I certainly agree you don't *need* JSON. However its a well understood
and reasonably well supported scheme that is already widely used for
managing resources.
A restricted subset of the AMQP type system would be better than nothing
(lest described types start appearing in 'standard' management
schemas!), but what's the value of that over just using something that
already exists and is simpler (yet sufficient)?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Comments on the AMQP management spec
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 11/04/14 11:42, Gordon Sim wrote:
> On 04/11/2014 10:48 AM, Fraser Adams wrote:
>> On 11/04/14 09:57, Gordon Sim wrote:
>>>
>>> In the spirit of keeping things as simple as possible, how about using
>>> JSON in the body? It is already in widespread use for modelling state
>>> for all sorts of things (including messaging brokers).
>>>
>> I've got some sympathy with that, though TBH I quite like the AMQP Maps
>> and Lists and those map pretty well to JSON, but with the added benefit
>> of a type system for stored primitives, so I'm not totally convinced.
>
> Personally I think the type system causes more problems than it
> solves. For the purposes of management you really only need strings
> and numbers (also arrays and associative arrays). All the extra
> 'richness' is just more scope for interoperability issues.
>
That's a fair point - don't I just know it trying to make intelligent
choices mapping numerical types from JavaScript :-) but ultimately
language bindings are going to have to have ways of dealing with that
anyway (unless we completely remove the AMQP 1.0 type system and replace
it with binary + string). In which case as far as I can see it the
problem space probably ought to fit into client side libraries as
opposed to the wire protocol - as said previously a JSON encode/decode
for Java/C++ would be nice, then client code could be written to use
JSON which would provide a decent AMQP mapping and not have any affect
on things that can serialise/deserialise this stuff fairly transparently.
"For the purposes of management you really only need strings and
numbers" might be reasonable, but you don't necessarily need JSON to
enforce this, perhaps the Management Spec being explicit about UTF
strings and double (I guess) for numbers and you can then map pretty
easily to/from JSON from a management perspective.
It's not that I'm arguing against JSON, my comments are more about what
should be transmitted on the wire vice what should be presented to users.
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Comments on the AMQP management spec
Posted by Gordon Sim <gs...@redhat.com>.
On 04/11/2014 10:48 AM, Fraser Adams wrote:
> On 11/04/14 09:57, Gordon Sim wrote:
>>
>> In the spirit of keeping things as simple as possible, how about using
>> JSON in the body? It is already in widespread use for modelling state
>> for all sorts of things (including messaging brokers).
>>
> I've got some sympathy with that, though TBH I quite like the AMQP Maps
> and Lists and those map pretty well to JSON, but with the added benefit
> of a type system for stored primitives, so I'm not totally convinced.
Personally I think the type system causes more problems than it solves.
For the purposes of management you really only need strings and numbers
(also arrays and associative arrays). All the extra 'richness' is just
more scope for interoperability issues.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Comments on the AMQP management spec
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 11/04/14 09:57, Gordon Sim wrote:
>
> In the spirit of keeping things as simple as possible, how about using
> JSON in the body? It is already in widespread use for modelling state
> for all sorts of things (including messaging brokers).
>
I've got some sympathy with that, though TBH I quite like the AMQP Maps
and Lists and those map pretty well to JSON, but with the added benefit
of a type system for stored primitives, so I'm not totally convinced.
If I'm really honest (and I've said this before so it won't be any
surprise) the best thing that could be done to simplify things from a
Java perspective it to use ObjectMessage to encode java.util.List and
java.util.Map. Being able to use standard containers and manipulation
thereof makes most of the pain go away, so I'm not convinced that the
extra encoding/decoding step into a JSON string is the right thing and
feels more of a cop-out/workaround
Another thing (and this might be moot in the context of the Management
Spec) is that it's not totally unreasonable to want to put binary data
into a Map - one of the things I experimented with a while back was the
possibility of using QMF method invocation as a "standardised" (I wish)
AMQP based remote method invocation framework. It actually had
everything I needed including a documented specification, mechanisms for
schema distribution - the works. I still think it was quite neat and
would have saved a ton of wheel reinvention. Anyway I used that to
retrieve large binary payloads as part of an experiment - you could of
course achieve this plenty of other ways, but as I say not having a
mechanism for binary contents is perhaps an unnecessary constraint on
users?
Ironically I've got a feeling using JSON actually *complicates* things
for many users. As you probably know I'm working on "productising"
JavaScript bindings to Proton Messenger this works in a very similar way
to the Python bindings where I can do something like:
message.setAddress(address);
message.setSubject(subject);
message.body = ['Rod', 'Jane', 'Freddy', {cat: true, donkey: 'hee
haw'}];
messenger.put(message);
So I can transparently serialise native JavaScript Arrays to AMQP list
and Object to AMQP map.
I'm quite looking forward to getting it finished so that I can mess
around with AMQP 1.0 Management in the most enjoyable way possible. I'm
not too keen on an extra JSON encoding/parsing step :-) though I could
(and would) clearly hide it in my put implementation.
My personal take on this is that using JSON might be nice from a client
perspective but that the best way to achieve that might be to add a JSON
encode/decode capability to the C++/Java tool sets and use AMQP maps and
lists on the wire so languages that can handle these things nicely
aren't penalised.
FWIW I suggested using Maps in the Management spec. because it was still
in draft so figured it might be easier to stop that nobbling Java than
to get Java to behave more sensibly :-D
While I'm on a roll...
Rob, could you do me a favour and make sure that the Management spec is
*absolutely clear and explicit* (referring to the relevant section in
the AMQP 1.0 spec would be good) that strings in management nodes and
manageable entities should be AMQP 1.0 UTF8 strings. I realise that it's
implicit in some of the wording, but I'm in no hurry to allow the
possibility that management nodes (or clients) might be sending C style
binary strings. I've been on the receiving end of too many
ClassCastExceptions of byte[] values in maps when I was reasonably
expecting String. I'd really like to be able to say that an application
is off spec if it sends binary strings and be able to clearly point to a
reference to say so #interoperability :-)
Cheers,
Frase
Re: Comments on the AMQP management spec
Posted by Gordon Sim <gs...@redhat.com>.
On 04/11/2014 09:35 AM, Fraser Adams wrote:
> whilst hyphens are indeed legal AMQP 1.0 the *are not* legal
> wrt. JMS message selectors, indeed they are not *actually* legal JMS
> application property names (I only realised that myself when I was
> trying to figure out the selector thing).
[...]
> there's no point making things hard for Java clients
Indeed. Keep things as simple as possible.
> On a related note previous working drafts of AMQP 1.0 Management had
> specified using lists as message bodies in quite a few places, which is
> perfectly legal AMQP (though as you have noted lists and maps as
> property values is not) but because lists are a pain for JMS I argued
> that was *probably* a bad idea.
In the spirit of keeping things as simple as possible, how about using
JSON in the body? It is already in widespread use for modelling state
for all sorts of things (including messaging brokers).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Comments on the AMQP management spec
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Hi Justin & Rob
I've replied to this on the TC comment list, but for completeness and
transparency here's what I said:
RE:
# Application properties and studlyCaps
The spec calls for studlyCaps for standard property names, whereas an
earlier draft had hyphenated names. FWIW, I prefer the latter, since
it’s in line with the core AMQP spec.
hyphens are an interesting case as I've found to my cost . You might
have seen my post to the qpid user list back in February entitled "A
write up of some AMQP 1.0 Experiments" during those experiments I
discovered some interesting behaviours with hyphenated names and message
selectors:
In precis whilst hyphens are indeed legal AMQP 1.0 the *are not* legal
wrt. JMS message selectors, indeed they are not *actually* legal JMS
application property names (I only realised that myself when I was
trying to figure out the selector thing). It turns out that JMS
properties have to have names that are valid Java identifiers.
Now technically that might be a moot point from the perspective of an
AMQP Specification - after all if it's legal AMQP.... but there's no
point making things hard for Java clients and it also doesn't seem great
to have one AMQP Specification specifying a form of properties that
would be illegal somewhere else:
https://svn.apache.org/repos/asf/qpid/trunk/qpid/specs/apache-filters.xml#section-legacy-amqp
Explaining the reason for the choice might be worthwhile though.
On a related note previous working drafts of AMQP 1.0 Management had
specified using lists as message bodies in quite a few places, which is
perfectly legal AMQP (though as you have noted lists and maps as
property values is not) but because lists are a pain for JMS I argued
that was *probably* a bad idea. I think that you do more python things
than Java don't you? For python, C++ and pretty much everything else
lists are fine, but for Java aaaarrrgggg!!
Re: # Ping
That's an interesting one, something like that would be useful, but I
personally don't think a simple ping to a $management node scales, so
that would be a very restrictive solution in a large topology of
containers or where there are lots of clients. I included the following
in an email to Rob but it was quite a long exchange and it may have been
a bit TL;DR :-) but here goes.....
One other thing that I keep forgetting to mention; in QMF2 Agents
broadcast heartbeat messages periodically to a topic so clients can
identify if Agents have gone away. I guess that this is perhaps more
useful in QMF2 where Agents aren't *necessarily* co-located with the
Broker, but it has got me thinking. In AMQP 1.0 Management I guess that
the $management node is likely to be pretty coupled to the Container's
lifecycle such that a client could probably infer that the $management
node is gone based on interception of a connection failure (e.g. via
something like a Connection Exception Listener) but a couple of use
cases fall out of that sort of scope:
1) What about Management nodes other than $management? It's possible
that the Container and $management might be available, but some other
Management node a user is interested in might possibly die
asynchronously for whatever reason (a simple use case might be one
client DEREGISTERing a Management Node that another client happens to be
using).
2) What about Message-oriented (as opposed to Connection-oriented) APIs
such as Messenger, with those the fact of Connection closing wouldn't
necessarily register until the next time a request was made.
Clearly a Management client could figure this out by some sort of
polling - and that might be better than needing a topic to broadcast a
heartbeat (though it doesn't scale as well if there are lots of clients)
but if polling is the expected way to accomplish this then I'd say that
the Management Specification needs to specify a standard Management node
operation to achieve this (perhaps STATUS ?). For someone designing a
decoupled client (say a WebApp) being able to determine the status of
Management nodes and provide feedback to the user is pretty useful.
Regards,
Frase
On 11/04/14 00:07, Rob Godfrey wrote:
> Hi Justin,
>
> can you send these comments to the public TC comment list (see [1]) so they
> can be incorporated in the discussions there on any updates to the spec (in
> your case - since you are employed by an organisation who is a member of
> OASIS and have voting members on the TC - this may not be strictly
> required... however it seems like good form :-) ).
>
> I'll try to respond on the technical points (later) in the morning.
>
> -- Rob
>
> [1] https://www.oasis-open.org/committees/comments/index.php?wg_abbrev=amqp
>
>
> On 11 April 2014 00:20, Justin Ross <jr...@apache.org> wrote:
>
>> Hi, everyone. I've been using the draft management spec to develop some
>> command line tools[1], and I've got some questions and comments. These
>> comments are based on working draft 8 of the spec[2].
>>
>> Thanks,
>> Justin
>>
>> [1] https://github.com/ssorj/bailey (Watch out! It's a work in progress.)
>> [2]
>>
>> https://www.oasis-open.org/committees/document.php?document_id=52425&wg_abbrev=amqp
>>
>> ---
>>
>> # Use of AMQP application-properties
>>
>> Given that values in AMQP 1.0 application properties cannot be lists or
>> maps (yikes--I didn't know that), I feel it's inadvisable to use
>> application properties to define requests. We'd be better off using
>> properties in the message body. Otherwise, we'll have to define standard
>> string encodings in every instance where we want to use a non-scalar type.
>>
>> # Application properties and studlyCaps
>>
>> The spec calls for studlyCaps for standard property names, whereas an
>> earlier draft had hyphenated names. FWIW, I prefer the latter, since it's
>> in line with the core AMQP spec.
>>
>> # QUERY and pagination
>>
>> There doesn't appear to be a way to query for the total count without
>> fetching all the values. Without that, you can't build a data efficient
>> page-navigation UI that offers links to each page (or just the last page).
>>
>> Requests using offset and count should perhaps instead use offset and
>> "limit". That's more familiar to those who have used SQL, and it correctly
>> signals that the number returned may be less than the number requested. In
>> my view, the request should use "limit", and the response should use
>> "count".
>>
>> Also, if we're doing paging we need to do sorting as well. Otherwise, A
>> client UI cannot build a sortable paginated table without pulling down
>> everything.
>>
>> # Request-response
>>
>> There's a note in section 4 that multiple response messages may be produced
>> for a single request. How should a client determine whether the response
>> is complete?
>>
>> # GET-MGMT-NODES
>>
>> The text says the response message must contain "a list of addresses of
>> other Management Nodes". Perhaps this should more explicitly prohibit
>> listing the currently-in-communication management node (or drop the word
>> "other").
>>
>> # [DE]REGISTER
>>
>> What is the intended use of this? That is, what new behavior do you get
>> when something is registered?
>>
>> # Ping
>>
>> I'd like to see a standard PING or ECHO operation on $management. I can
>> simulate it by using an arbitrary operation, but that seems less than
>> straightforward for implementors.
>>
>> # Clerical stuff
>>
>> 3.3.3.1: Awkward phrasing: "so if any of the changes cannot be applied,
>> the
>> entire operation should not be applied and to multiple values changed this
>> MUST result in a failure response"
>> 3.4.1: Typo: "this operation supports pagination <though> which a request"
>> 3.4.1: Awkward phrasing: "A result set of size N <can be considered to
>> containing elements>"
>>
>> # Odds and ends
>>
>> - 2.3: Should this reference the AMQP definition of a "node"?
>> - 3.1: It would be nice to mention that locales is a comma-separated list.
>> The RFC mentions it, but you have to dig down a bit.
>> - 3.3.1.2: What is a good example of a generic value made specific in an
>> UPDATE response?
>> - 3.3.3: Is the lack of attribute append a problem? Right now, there is
>> only replacing list values, and dueling actors could mean lost state.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Comments on the AMQP management spec
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Justin,
can you send these comments to the public TC comment list (see [1]) so they
can be incorporated in the discussions there on any updates to the spec (in
your case - since you are employed by an organisation who is a member of
OASIS and have voting members on the TC - this may not be strictly
required... however it seems like good form :-) ).
I'll try to respond on the technical points (later) in the morning.
-- Rob
[1] https://www.oasis-open.org/committees/comments/index.php?wg_abbrev=amqp
On 11 April 2014 00:20, Justin Ross <jr...@apache.org> wrote:
> Hi, everyone. I've been using the draft management spec to develop some
> command line tools[1], and I've got some questions and comments. These
> comments are based on working draft 8 of the spec[2].
>
> Thanks,
> Justin
>
> [1] https://github.com/ssorj/bailey (Watch out! It's a work in progress.)
> [2]
>
> https://www.oasis-open.org/committees/document.php?document_id=52425&wg_abbrev=amqp
>
> ---
>
> # Use of AMQP application-properties
>
> Given that values in AMQP 1.0 application properties cannot be lists or
> maps (yikes--I didn't know that), I feel it's inadvisable to use
> application properties to define requests. We'd be better off using
> properties in the message body. Otherwise, we'll have to define standard
> string encodings in every instance where we want to use a non-scalar type.
>
> # Application properties and studlyCaps
>
> The spec calls for studlyCaps for standard property names, whereas an
> earlier draft had hyphenated names. FWIW, I prefer the latter, since it's
> in line with the core AMQP spec.
>
> # QUERY and pagination
>
> There doesn't appear to be a way to query for the total count without
> fetching all the values. Without that, you can't build a data efficient
> page-navigation UI that offers links to each page (or just the last page).
>
> Requests using offset and count should perhaps instead use offset and
> "limit". That's more familiar to those who have used SQL, and it correctly
> signals that the number returned may be less than the number requested. In
> my view, the request should use "limit", and the response should use
> "count".
>
> Also, if we're doing paging we need to do sorting as well. Otherwise, A
> client UI cannot build a sortable paginated table without pulling down
> everything.
>
> # Request-response
>
> There's a note in section 4 that multiple response messages may be produced
> for a single request. How should a client determine whether the response
> is complete?
>
> # GET-MGMT-NODES
>
> The text says the response message must contain "a list of addresses of
> other Management Nodes". Perhaps this should more explicitly prohibit
> listing the currently-in-communication management node (or drop the word
> "other").
>
> # [DE]REGISTER
>
> What is the intended use of this? That is, what new behavior do you get
> when something is registered?
>
> # Ping
>
> I'd like to see a standard PING or ECHO operation on $management. I can
> simulate it by using an arbitrary operation, but that seems less than
> straightforward for implementors.
>
> # Clerical stuff
>
> 3.3.3.1: Awkward phrasing: "so if any of the changes cannot be applied,
> the
> entire operation should not be applied and to multiple values changed this
> MUST result in a failure response"
> 3.4.1: Typo: "this operation supports pagination <though> which a request"
> 3.4.1: Awkward phrasing: "A result set of size N <can be considered to
> containing elements>"
>
> # Odds and ends
>
> - 2.3: Should this reference the AMQP definition of a "node"?
> - 3.1: It would be nice to mention that locales is a comma-separated list.
> The RFC mentions it, but you have to dig down a bit.
> - 3.3.1.2: What is a good example of a generic value made specific in an
> UPDATE response?
> - 3.3.3: Is the lack of attribute append a problem? Right now, there is
> only replacing list values, and dueling actors could mean lost state.
>