You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Daniel Pocock <da...@pocock.pro> on 2017/04/03 09:18:39 UTC
Re: qpid-proton and rabbitmq (and Kamailio kazoo module)
On 30/03/17 13:25, Gordon Sim wrote:
> On 30/03/17 09:52, Daniel Pocock wrote:
>> Sending to RabbitMQ with qpid-proton python
>> -------------------------------------------
>>
>> I started with the simple_send.py[3] example and trimmed it down to send
>> a single string and then stop:
>>
>> def on_sendable(self, event):
>> if self.msg_body is not None:
>> print("on_sendable !")
>> print ("sending : %s" % (self.msg_body,))
>> msg = Message(body=self.msg_body)
>> event.sender.send(msg)
>> self.msg_body = None
>>
>> I found that the connection to RabbitMQ would be dropped and an error
>> would appear in the RabbitMQ log. I tried changing the body type to a
>> dict containing a string, e.g. {'my_body':'foo'} and then it would
>> successfully pass the message to RabbitMQ but then the receiver would
>> complain that it was a map and not a string.
>>
>> I found that the problem could be fixed by adding "inferred=True" to the
>> Message constructor:
>>
>> msg = Message(body=self.msg_body, inferred=True)
>
>
> The python proton library will encode a binary message body as a Data
> section if inferred is not set to true (otherwise as a binary value in a
> Value section). RabbitMQ does not support Data sections yet I believe.
>
Is there a bug or feature request for Data sections in RabbitMQ? I
couldn't find their bug tracker. I've opened a bug report[1] against
the Debian package.
> As Robbie mentioned, in python 2.x string literals are not always
> explicitly utf8 and are then treated as binary. You can either do u'my
> utf8' or import unicode_literals from __future__. If you do that then
> you should get a Value section regardless of the inferred property.
>
In my case the strings are coming from the command line, so now I change
the string to unicode like this:
_s = unicode(s, "utf-8")
and then:
a) I can send without inferred=True and the broker accepts the message
b) in the C++ receiver, the message is now received as a string and I
can use:
_json = proton::get<std::string>(m.body());
Could this be made more obvious in the examples perhaps, e.g.
helloworld.py could change from:
event.sender.send(Message(body="Hello World!"))
to
event.sender.send(Message(body=unicode("Hello World!", "utf-8")))
I realize that is a bit verbose and redundant when you already have
unicode_literals, but for people who are cutting and pasting from the
examples they will get up and running more quickly.
>> Below is a complete example of the error from the RabbitMQ log when
>> inferred=True is not present, the body in this case was the string "foo"
>>
>>
>> Receiving from RabbitMQ with qpid-proton C++
>> --------------------------------------------
>>
>> I looked at the receive example simple_recv.cpp[4] and other examples
>> and noticed code like this being used to access the message body as
>> std::string:
>>
>> proton::get<std::string>(m.body())
>>
>> However, that didn't work for me, it threw a conversion_error exception
>> "unexpected type, want: string got: binary" complaining that the body
>> was binary. I had to use something like this to convert my message body
>> to a std::string:
>>
>> proton::binary __b = proton::get<proton::binary>(m.body());
>> std::string _s = (std::string)__b;
>>
>> If I send a string from Python, should it appear as binary in the
>> receiver?
>
> As above it sounds like the message *is* actually being sent as a binary.
>
>> What is the suggested way to write a receiver that can handle
>> any arbitrary message that started as a string and may arrive in some
>> other format, especially if the message broker or client is changed at
>> some arbitrary time in the future?
>>
>> I also observed similar issues receiving messages that had been sent
>> into RabbitMQ by Kamailio's kazoo module[5], it is linked with the
>> librabbitmq client library.
>
> I'm assuming that is sending over AMQP 0-10. If so, how that is
> converted into AMQP 1.0 is dependent on the RabbitMQ implementation. It
> may well be always sent as a binary value. However hopefully it would
> have a content-type to indicate how to interpret the data (though
> structly speaking the AMQP 1.0 spec disallows that).
>
The kazoo module source code is on Github[2], I had a quick look at it
and it isn't immediately obvious to me which AMQP version it uses or
which data type it is using for the message body. Is anybody more
familiar with AMQP C client programming able to spot any keywords in
that code that suggest what it is doing? I've added the kazoo module
developers on CC in case they can comment on this, it would be useful to
answer this in the module's README file.
Regards,
Daniel
1. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859399
2. https://github.com/sipwise/kamailio/tree/master/modules/kazoo
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-proton and rabbitmq (and Kamailio kazoo module)
Posted by Gordon Sim <gs...@redhat.com>.
On 03/04/17 17:26, Daniel Pocock wrote:
>
>
> On 03/04/17 17:56, Luis Azedo wrote:
>> kazoo module uses whatever rabbitmq supports (AMQP 0-9-1)
>>
>> https://www.rabbitmq.com/protocol.html
>>
>>
>> content type is
>>
>> props._flags = AMQP_BASIC_CONTENT_TYPE_FLAG;
>> props.content_type = amqp_cstring_bytes("application/json");
>>
>> this is enforced in publishing and consuming
>>
>> and the proper url for the kazoo module would
>> be https://github.com/kamailio/kamailio/tree/master/src/modules/kazoo
>>
>> kazoo module is built with https://github.com/alanxz/rabbitmq-c
>>
>> if you're using python, you may want to
>> try https://pika.readthedocs.io/en/0.10.0/ instead
>>
>
> Hi Luis,
>
> Thanks for the feedback
>
> When I receive the messages using the Qpid Proton C++ client library (it
> talks to RabbitMQ using AMQP 1.0), it receives binary messages.
>
> Does your code send them in to RabbitMQ using type binary or string?
>
> If you submit the body as a string, do you believe RabbitMQ server is
> changing the type from string to binary?
In AMQP 0-9-1, the body content of messages is always 'binary' i.e.
untyped bytes. There is then a content-type property that lets you
determine how to handle it (as for HTTP for example).
I suspect that RabbitMQ converts such messages to binary when converting
to AMQP 1.0. If it retains the content-type property you could use that
on the receiver side to convert to a string where appropriate.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-proton and rabbitmq (and Kamailio kazoo module)
Posted by Daniel Pocock <da...@pocock.pro>.
On 03/04/17 17:56, Luis Azedo wrote:
> kazoo module uses whatever rabbitmq supports (AMQP 0-9-1)
>
> https://www.rabbitmq.com/protocol.html
>
>
> content type is
>
> props._flags = AMQP_BASIC_CONTENT_TYPE_FLAG;
> props.content_type = amqp_cstring_bytes("application/json");
>
> this is enforced in publishing and consuming
>
> and the proper url for the kazoo module would
> be https://github.com/kamailio/kamailio/tree/master/src/modules/kazoo
>
> kazoo module is built with https://github.com/alanxz/rabbitmq-c
>
> if you're using python, you may want to
> try https://pika.readthedocs.io/en/0.10.0/ instead
>
Hi Luis,
Thanks for the feedback
When I receive the messages using the Qpid Proton C++ client library (it
talks to RabbitMQ using AMQP 1.0), it receives binary messages.
Does your code send them in to RabbitMQ using type binary or string?
If you submit the body as a string, do you believe RabbitMQ server is
changing the type from string to binary?
Regards,
Daniel
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-proton and rabbitmq (and Kamailio kazoo module)
Posted by Luis Azedo <lu...@2600hz.com>.
kazoo module uses whatever rabbitmq supports (AMQP 0-9-1)
https://www.rabbitmq.com/protocol.html
content type is
props._flags = AMQP_BASIC_CONTENT_TYPE_FLAG;
props.content_type = amqp_cstring_bytes("application/json");
this is enforced in publishing and consuming
and the proper url for the kazoo module would be https://github.com/kamailio/kamailio/tree/master/src/modules/kazoo
kazoo module is built with https://github.com/alanxz/rabbitmq-c
if you're using python, you may want to try https://pika.readthedocs.io/en/0.10.0/ instead
hope this helps
Cheers
________________________________
From: Daniel Pocock <da...@pocock.pro>
Sent: Monday, April 3, 2017 10:18:39 AM
To: users@qpid.apache.org
Cc: Engineering; Luis Azedo; sr-dev@lists.sip-router.org
Subject: Re: qpid-proton and rabbitmq (and Kamailio kazoo module)
On 30/03/17 13:25, Gordon Sim wrote:
> On 30/03/17 09:52, Daniel Pocock wrote:
>> Sending to RabbitMQ with qpid-proton python
>> -------------------------------------------
>>
>> I started with the simple_send.py[3] example and trimmed it down to send
>> a single string and then stop:
>>
>> def on_sendable(self, event):
>> if self.msg_body is not None:
>> print("on_sendable !")
>> print ("sending : %s" % (self.msg_body,))
>> msg = Message(body=self.msg_body)
>> event.sender.send(msg)
>> self.msg_body = None
>>
>> I found that the connection to RabbitMQ would be dropped and an error
>> would appear in the RabbitMQ log. I tried changing the body type to a
>> dict containing a string, e.g. {'my_body':'foo'} and then it would
>> successfully pass the message to RabbitMQ but then the receiver would
>> complain that it was a map and not a string.
>>
>> I found that the problem could be fixed by adding "inferred=True" to the
>> Message constructor:
>>
>> msg = Message(body=self.msg_body, inferred=True)
>
>
> The python proton library will encode a binary message body as a Data
> section if inferred is not set to true (otherwise as a binary value in a
> Value section). RabbitMQ does not support Data sections yet I believe.
>
Is there a bug or feature request for Data sections in RabbitMQ? I
couldn't find their bug tracker. I've opened a bug report[1] against
the Debian package.
> As Robbie mentioned, in python 2.x string literals are not always
> explicitly utf8 and are then treated as binary. You can either do u'my
> utf8' or import unicode_literals from __future__. If you do that then
> you should get a Value section regardless of the inferred property.
>
In my case the strings are coming from the command line, so now I change
the string to unicode like this:
_s = unicode(s, "utf-8")
and then:
a) I can send without inferred=True and the broker accepts the message
b) in the C++ receiver, the message is now received as a string and I
can use:
_json = proton::get<std::string>(m.body());
Could this be made more obvious in the examples perhaps, e.g.
helloworld.py could change from:
event.sender.send(Message(body="Hello World!"))
to
event.sender.send(Message(body=unicode("Hello World!", "utf-8")))
I realize that is a bit verbose and redundant when you already have
unicode_literals, but for people who are cutting and pasting from the
examples they will get up and running more quickly.
>> Below is a complete example of the error from the RabbitMQ log when
>> inferred=True is not present, the body in this case was the string "foo"
>>
>>
>> Receiving from RabbitMQ with qpid-proton C++
>> --------------------------------------------
>>
>> I looked at the receive example simple_recv.cpp[4] and other examples
>> and noticed code like this being used to access the message body as
>> std::string:
>>
>> proton::get<std::string>(m.body())
>>
>> However, that didn't work for me, it threw a conversion_error exception
>> "unexpected type, want: string got: binary" complaining that the body
>> was binary. I had to use something like this to convert my message body
>> to a std::string:
>>
>> proton::binary __b = proton::get<proton::binary>(m.body());
>> std::string _s = (std::string)__b;
>>
>> If I send a string from Python, should it appear as binary in the
>> receiver?
>
> As above it sounds like the message *is* actually being sent as a binary.
>
>> What is the suggested way to write a receiver that can handle
>> any arbitrary message that started as a string and may arrive in some
>> other format, especially if the message broker or client is changed at
>> some arbitrary time in the future?
>>
>> I also observed similar issues receiving messages that had been sent
>> into RabbitMQ by Kamailio's kazoo module[5], it is linked with the
>> librabbitmq client library.
>
> I'm assuming that is sending over AMQP 0-10. If so, how that is
> converted into AMQP 1.0 is dependent on the RabbitMQ implementation. It
> may well be always sent as a binary value. However hopefully it would
> have a content-type to indicate how to interpret the data (though
> structly speaking the AMQP 1.0 spec disallows that).
>
The kazoo module source code is on Github[2], I had a quick look at it
and it isn't immediately obvious to me which AMQP version it uses or
which data type it is using for the message body. Is anybody more
familiar with AMQP C client programming able to spot any keywords in
that code that suggest what it is doing? I've added the kazoo module
developers on CC in case they can comment on this, it would be useful to
answer this in the module's README file.
Regards,
Daniel
1. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859399
2. https://github.com/sipwise/kamailio/tree/master/modules/kazoo