AMQP map ordering guarantees on the C++ binding

[Kim van der Riet <ki...@redhat.com>]

When creating an AMQP map body, using

std::map<proton::value, proton::value> map;

to represent the map, and once its values have been set,

proton::message msg;
msg.body(map);

is a convenient way to set the message body. However, in practice, I am 
finding this approach has one problem - I cannot guarantee the key/value 
ordering within the map.

On the receiving end, if I don't know the map keys in advance, I need to 
use a similar approach:

std::map<proton::value, proton::value> map;
proton::get(msg.body(), map);

which puts the map into a std::map where I can iterate through keys, 
etc. But as soon as I do this, I have lost the ordering guarantees of 
the original message, as the map will iterate through the keys in an 
internal order unrelated to the message ordering.

What is the correct way to both set a proton::value as a map in such a 
way that the ordering is preserved when I both set the values and read 
them afterward without knowledge of the keys?

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: AMQP map ordering guarantees on the C++ binding

[Alan Conway <ac...@redhat.com>]

In C++ you can convert a proton::value containing an AMQP map to/from
std::deque, std::vector, std::list, std::forward_list or  std::deque of
std::pair<proton::value, proton::value> That preserves order in decoding
and gives you control of the order for encoding.

You need to #include <proton/codec/vector.hpp> or whichever to enable the
conversion - have a look in the proton/codec include dir to see all the
conversions we have. Once you've done the #include you can insert/extract
those types like any other type.

This is documented at http://qpid.apache.org/releases/qpid-proton-0.21.0/
proton/cpp/api/types_page.html is very thin, I've raised a JIRA to improve
it: https://issues.apache.org/jira/browse/PROTON-1795

In practice I expect it will be very rare that anybody cares about map
order, but sadly the AMQP spec does declare it to be significant. In formal
terms an AMQP map is not semantically a map at all, but a sequence of
alternating key/value pairs:

<spec>
A map is encoded as a compound value where the constituent elements form
alternating key value pairs.
item 0 item 1 item n-1 item n
+-------+-------+----+---------+---------+
| key 1 | val 1 | .. | key n/2 | val n/2 |
+-------+-------+----+---------+---------+
Figure 1.20: Layout of Map Encoding
Map encodings MUST contain an even number of items (i.e. an equal number of
keys and values). A map in
which there exist two identical key values is invalid. Unless known to be
otherwise, maps MUST be considered
to be ordered, that is, the order of the key-value pairs is semantically
important and two maps which are different
only in the order in which their key-value pairs are encoded are not equal.
<spec>

So to be fully compliant and interoperable we MUST be able to control/view
the encoded order of a map. For apps using proton in a sane way we can
assume it is "known to be otherwise" but we are supposed to interop with
arbitrary AMQP endpoints about which we know nothing. However it is
probably a low priority to test/fix compared to other complex type testing.
An application that actually did depend on the ordering of a map would be
theoretically compliant but pragmatically insane.



On Fri, Mar 16, 2018 at 11:18 AM, Kim van der Riet <ki...@redhat.com>
wrote:

> On 03/15/2018 05:23 PM, Andrew Stitcher wrote:
>
>> On Thu, 2018-03-15 at 15:54 -0400, Kim van der Riet wrote:
>>
>>> When creating an AMQP map body, using
>>>
>>> std::map<proton::value, proton::value> map;
>>>
>>> to represent the map, and once its values have been set,
>>>
>>> proton::message msg;
>>> msg.body(map);
>>>
>>> is a convenient way to set the message body. However, in practice, I
>>> am
>>> finding this approach has one problem - I cannot guarantee the
>>> key/value
>>> ordering within the map.
>>>
>>
>> Why do want to do this?
>>
> This is a part of the AMQP client interoperability testing. Part of the
> test must ensure that the map is not re-ordered in any way by the clients
> or broker.
>
>>
>>
>>> On the receiving end, if I don't know the map keys in advance, I need
>>> to
>>> use a similar approach:
>>>
>>> std::map<proton::value, proton::value> map;
>>> proton::get(msg.body(), map);
>>>
>>> which puts the map into a std::map where I can iterate through keys,
>>> etc. But as soon as I do this, I have lost the ordering guarantees
>>> of
>>> the original message, as the map will iterate through the keys in an
>>> internal order unrelated to the message ordering.
>>>
>>> What is the correct way to both set a proton::value as a map in such
>>> a
>>> way that the ordering is preserved when I both set the values and
>>> read
>>> them afterward without knowledge of the keys?
>>>
>>
>> I'm not sure what you are trying to accomplish, perhaps you can explain
>>   and that would make this easier to answer.
>>
>
> For interoperability test purposes, I need to create and send an AMQP
> message containing an AMQP map type. The map needs to have a well-defined
> sequence of keys/values. On receipt, the map must be retrieved in such a
> way that the order as sent on the wire is not lost, and once retrieved,
> compared with the original map.
>
> On message creation, I assume that using the proton::map::put(k,v) will
> ensure that the elements will be inserted into the map in the order this
> function is called. However, retrieving the map from a message in a way
> that guarantees the order of the map eludes me. There is a
> proton::map::get(k) method, but that assumes you know the keys. The
> receivers have no knowledge of the map contents or keys, so it must somehow
> get an ordered list of keys. I cannot see how to do that. It seems to me
> that the API needs something like
>
> std::vector<proton::value> proton::map::get_keys();
>
> which would provide an ordered key list. Then using proton::map::get(k)
> makes sense.
>
> Without this, there is no way to test the ordering guarantees required by
> the AMQP 1.0 spec.
>
>
>> Essentially though the AMQP concept of a map and language binding
>> concepts of maps differ.
>>
>> They are just the closest most convenient things that allow users to
>> accomplish what they most likely want to accomplish: That is to send an
>> application level map/dictionary and receive it in a perhaps different
>> language environment with the same meaning (more-or-less).
>>
>> As it happens std::map actually does have a well defined order (it is
>> always ordered by the "less-than" relation on its keys) and I believe
>> that the Qpid C++ binding coder will transcribe that into the AMQP map
>> - this is not one of our tests though so it is entirely possible the
>> iterator used goes backwards instead!
>>
>
> How is the "less-than" relation set for proton::value objects? If there
> were a method to set the order through calling a setter function, that
> would make life a lot easier.
>
>
>
>> However the map binding in other languages is much more likely to use
>> the hash of the keys in the data structure and so there will be no well
>> defined order for them.
>>
>> There are other "features" of AMQP maps that just aren't directly
>> transcribeable to language features - especially if there are repeated
>> identical keys in an AMQP map.
>>
>> Hope that helps
>>
>> Andrew
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: AMQP map ordering guarantees on the C++ binding

[Kim van der Riet <ki...@redhat.com>]

On 03/15/2018 05:23 PM, Andrew Stitcher wrote:
> On Thu, 2018-03-15 at 15:54 -0400, Kim van der Riet wrote:
>> When creating an AMQP map body, using
>>
>> std::map<proton::value, proton::value> map;
>>
>> to represent the map, and once its values have been set,
>>
>> proton::message msg;
>> msg.body(map);
>>
>> is a convenient way to set the message body. However, in practice, I
>> am
>> finding this approach has one problem - I cannot guarantee the
>> key/value
>> ordering within the map.
> 
> Why do want to do this?
This is a part of the AMQP client interoperability testing. Part of the 
test must ensure that the map is not re-ordered in any way by the 
clients or broker.
> 
>>
>> On the receiving end, if I don't know the map keys in advance, I need
>> to
>> use a similar approach:
>>
>> std::map<proton::value, proton::value> map;
>> proton::get(msg.body(), map);
>>
>> which puts the map into a std::map where I can iterate through keys,
>> etc. But as soon as I do this, I have lost the ordering guarantees
>> of
>> the original message, as the map will iterate through the keys in an
>> internal order unrelated to the message ordering.
>>
>> What is the correct way to both set a proton::value as a map in such
>> a
>> way that the ordering is preserved when I both set the values and
>> read
>> them afterward without knowledge of the keys?
> 
> I'm not sure what you are trying to accomplish, perhaps you can explain
>   and that would make this easier to answer.

For interoperability test purposes, I need to create and send an AMQP 
message containing an AMQP map type. The map needs to have a 
well-defined sequence of keys/values. On receipt, the map must be 
retrieved in such a way that the order as sent on the wire is not lost, 
and once retrieved, compared with the original map.

On message creation, I assume that using the proton::map::put(k,v) will 
ensure that the elements will be inserted into the map in the order this 
function is called. However, retrieving the map from a message in a way 
that guarantees the order of the map eludes me. There is a 
proton::map::get(k) method, but that assumes you know the keys. The 
receivers have no knowledge of the map contents or keys, so it must 
somehow get an ordered list of keys. I cannot see how to do that. It 
seems to me that the API needs something like

std::vector<proton::value> proton::map::get_keys();

which would provide an ordered key list. Then using proton::map::get(k) 
makes sense.

Without this, there is no way to test the ordering guarantees required 
by the AMQP 1.0 spec.

> 
> Essentially though the AMQP concept of a map and language binding
> concepts of maps differ.
> 
> They are just the closest most convenient things that allow users to
> accomplish what they most likely want to accomplish: That is to send an
> application level map/dictionary and receive it in a perhaps different
> language environment with the same meaning (more-or-less).
> 
> As it happens std::map actually does have a well defined order (it is
> always ordered by the "less-than" relation on its keys) and I believe
> that the Qpid C++ binding coder will transcribe that into the AMQP map
> - this is not one of our tests though so it is entirely possible the
> iterator used goes backwards instead!

How is the "less-than" relation set for proton::value objects? If there 
were a method to set the order through calling a setter function, that 
would make life a lot easier.

> 
> However the map binding in other languages is much more likely to use
> the hash of the keys in the data structure and so there will be no well
> defined order for them.
> 
> There are other "features" of AMQP maps that just aren't directly
> transcribeable to language features - especially if there are repeated
> identical keys in an AMQP map.
> 
> Hope that helps
> 
> Andrew
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: AMQP map ordering guarantees on the C++ binding

[Andrew Stitcher <as...@apache.org>]

On Thu, 2018-03-15 at 15:54 -0400, Kim van der Riet wrote:
> When creating an AMQP map body, using
> 
> std::map<proton::value, proton::value> map;
> 
> to represent the map, and once its values have been set,
> 
> proton::message msg;
> msg.body(map);
> 
> is a convenient way to set the message body. However, in practice, I
> am 
> finding this approach has one problem - I cannot guarantee the
> key/value 
> ordering within the map.

Why do want to do this?

> 
> On the receiving end, if I don't know the map keys in advance, I need
> to 
> use a similar approach:
> 
> std::map<proton::value, proton::value> map;
> proton::get(msg.body(), map);
> 
> which puts the map into a std::map where I can iterate through keys, 
> etc. But as soon as I do this, I have lost the ordering guarantees
> of 
> the original message, as the map will iterate through the keys in an 
> internal order unrelated to the message ordering.
> 
> What is the correct way to both set a proton::value as a map in such
> a 
> way that the ordering is preserved when I both set the values and
> read 
> them afterward without knowledge of the keys?

I'm not sure what you are trying to accomplish, perhaps you can explain
 and that would make this easier to answer.

Essentially though the AMQP concept of a map and language binding
concepts of maps differ.

They are just the closest most convenient things that allow users to
accomplish what they most likely want to accomplish: That is to send an
application level map/dictionary and receive it in a perhaps different
language environment with the same meaning (more-or-less).

As it happens std::map actually does have a well defined order (it is
always ordered by the "less-than" relation on its keys) and I believe
that the Qpid C++ binding coder will transcribe that into the AMQP map
- this is not one of our tests though so it is entirely possible the
iterator used goes backwards instead!

However the map binding in other languages is much more likely to use
the hash of the keys in the data structure and so there will be no well
defined order for them.

There are other "features" of AMQP maps that just aren't directly
transcribeable to language features - especially if there are repeated
identical keys in an AMQP map.

Hope that helps

Andrew


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org