You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Fraser Adams <fr...@blueyonder.co.uk> on 2011/09/16 14:56:32 UTC
Significant Headers Exchange interoperability issues noticed - was
Re: JMS client skips string message properties
Hi All,
I was playing around looking at the symptoms Jiri described and I
decided to check out some nagging concerns I've had with the headers
exchange.
I'm surprised I've not noticed this before, but I've mainly been looking
at java and haven't focused on interoperability issues as much as I
perhaps should have.
So I've tried setting up producers and consumers in c++, java and perl
with roughly equivalent behaviour. I use an address on each as follows:
C++:
string address = "test; {create: receiver, node: {x-declare: {arguments:
{'qpid.policy_type': ring, 'qpid.max_size': 500000000}}, x-bindings:
[{exchange: 'amq.match', queue: 'test', key: 'data1', arguments:
{x-match: all, data-service: amqp-delivery, item-owner: fadams}}]}}";
Java (via a JNDI file):
destination.subscribedAddress1 = test; {create: receiver, node:
{x-declare: {arguments: {'qpid.policy_type': ring, 'qpid.max_size':
500000000}}, x-bindings: [{exchange: 'amq.match', queue: 'test', key:
'data1', arguments: {x-match: all, data-service: amqp-delivery,
item-owner: fadams}}]}}
Perl:
my $address = "test; {create: receiver, node: {x-declare: {arguments:
{'qpid.policy_type': ring, 'qpid.max_size': 500000000}}, x-bindings:
[{exchange: 'amq.match', queue: 'test', key: 'data1', arguments:
{x-match: all, data-service: amqp-delivery, item-owner: fadams}}]}}";
I had producers writing to amq.match and I found:
perl producer -> perl consumer OK
perl producer -> c++ consumer OK
perl producer -> java consumer FAILED
c++ producer -> perl consumer OK
c++ producer -> c++ consumer OK
c++ producer -> java consumer FAILED
java producer -> perl consumer FAILED
java producer -> c++ consumer FAILED
java producer -> java consumer OK
I know the perl one is a bit of a red herring as it's essentially a
wrapper round qpid::messaging but I included for completeness. My tests
were carried out using the 0.10 C++ broker using the qpid::messaging API.
So the thread preceding this has discussed the UTF-8 issues with respect
to variant types but this is perhaps more insidious. I couldn't quite
figure it out until I looked using a qpid-config I'd patched so I could
see headers bindings.
For C++ and perl I get:
bind [data1] => amq.match {u'item-owner': 'fadams', u'data-service':
'amqp-delivery', u'x-match': 'all'}
For java I get:
bind [data1] => amq.match {u'item-owner': u'fadams', u'data-service':
u'amqp-delivery', u'x-match': u'all'}
So basically for Java the address string pushes Unicode values onto the
binding, but for C++/perl the address string pushes (I guess) ASCII
values onto the binding.
This is a pretty big concern for me. But one thing is even weirder; in
my organisation we've got a mixture of c++ and java producers and
consumers using the headers exchange ,but so far (touch wood) things
*seem* to be working. However as it happens the C++ producers and
consumers are using the qpid::client API, so are likely to be binding at
a lower level rather than using address strings.
I'm trying to persuade the developers to switch to qpid::messaging but
now I'm worried this could cause interoperability issues.
I have to confess that this is really pretty frustrating. I seem to keep
getting bitten by string incompatibilities, which I can mostly mitigate
at the client code level, but problems at the exchange are more painful.
I keep on arguing that property strings should default to Unicode to
ensure interoperability and this convinces me more. Nobody seems to be
biting on that though - is it just me who's been driven nuts by this??
Is there any way that I can force the address string in C++/perl to be
treated as Unicode in
Receiver receiver = session.createReceiver(address);
Interoperability is really important to me.
Cheers
Frase
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Authentication newbie help sought - I'm afraid I'm failing at the
first hurdle :-(
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Hi all,
I thought I'd have an initial play with authentication, but I'm afraid
that I seem to be failing at the first hurdle.
So what I've done so far is:
I knew that there are potential issues with permissions with the
qpidd.sasldb so my first step was to copy etc/sasl2/qpidd.conf and
qpidd.sasldb to my home directory (just to make things easier while I'm
playing). I modified qpidd.conf with
sasldb_path: /home/fadams/qpidd.sasldb
I checked
sasldblistusers2 -f /home/fadams/qpidd.sasldb
and got (as expected)
guest@QPID: userPassword
and was able to add other users using
saslpasswd2 -f /home/fadams/qpidd.sasldb -u QPID fadams
So I started qpidd as myself with:
qpidd --sasl-config /home/fadams/qpidd.conf -t
And in the trace I got:
2011-10-04 18:57:59 info SASL: config path set to /home/fadams/qpidd.conf
2011-10-04 18:57:59 info SASL enabled
2011-10-04 18:57:59 notice Listening on TCP port 5672
2011-10-04 18:57:59 info Policy file not specified. ACL Disabled, no ACL
checking being done!
2011-10-04 18:57:59 notice Broker running
2011-10-04 18:58:04 debug RECV [127.0.0.1:5672-127.0.0.1:35444] INIT(0-10)
2011-10-04 18:58:04 debug External ssf=0 and auth=
2011-10-04 18:58:04 debug min_ssf: 0, max_ssf: 256, external_ssf: 0
2011-10-04 18:58:04 info SASL: Mechanism list: NTLM CRAM-MD5 LOGIN
DIGEST-MD5 ANONYMOUS PLAIN
Which looked OK to me.
However I then tried to connect with a Java consumer using a fairly
basic connection URL
connectionfactory.ConnectionFactory =
amqp://guest:guest@clientid/test?brokerlist='tcp://localhost:5672'
Which failed with "session creation failed"
And a broker trace of:
2011-10-04 18:58:04 trace SENT [127.0.0.1:5672-127.0.0.1:35444]:
Frame[BEbe; channel=0; {ConnectionStartBody:
server-properties={qpid.federation_tag:V2:36:str16(04cb2a36-ccaa-4762-9e9a-56329c267085)};
mechanisms=str16{V2:4:str16(NTLM), V2:8:str16(CRAM-MD5),
V2:5:str16(LOGIN), V2:10:str16(DIGEST-MD5), V2:9:str16(ANONYMOUS),
V2:5:str16(PLAIN)}; locales=str16{V2:5:str16(en_US)}; }]
2011-10-04 18:58:04 trace RECV [127.0.0.1:5672-127.0.0.1:35444]:
Frame[BEbe; channel=0; {ConnectionStartOkBody:
client-properties={clientName:V2:8:str16(clientid),qpid.client_pid:F4:int32(7032),qpid.client_process:V2:16:str16(Qpid
Java Client),qpid.session_flow:F4:int32(1)}; mechanism=PLAIN;
response=xxxxxx; }]
2011-10-04 18:58:04 debug SASL: Starting authentication with mechanism:
PLAIN
2011-10-04 18:58:04 info SASL: Authentication failed for
guest@QPID:SASL(-13): user not found: Password verification failed
2011-10-04 18:58:04 debug Exception constructed: Authentication failed
2011-10-04 18:58:04 debug SEND raiseEvent (v1)
class=org.apache.qpid.broker.clientConnectFail
2011-10-04 18:58:04 debug SEND raiseEvent (v2)
class=org.apache.qpid.broker.clientConnectFail
2011-10-04 18:58:04 trace SENT [127.0.0.1:5672-127.0.0.1:35444]:
Frame[BEbe; channel=0; {ConnectionCloseBody: reply-code=320;
reply-text=connection-forced: Authentication failed; }]
2011-10-04 18:58:04 trace RECV [127.0.0.1:5672-127.0.0.1:35444]:
Frame[BEbe; channel=0; {ConnectionCloseOkBody: }]
2011-10-04 18:58:04 debug DISCONNECTED [127.0.0.1:5672-127.0.0.1:35444]
I also tried explicitly setting the realm using:
qpidd --sasl-config /home/fadams/qpidd.conf --realm QPID -t
but that was equally unsuccessful.
Finally as much out of desperation as anything I tried:
sudo qpidd -t
which clearly should have picked up the default stuff in the default
qpidd.sasldb locations and clearly would have the correct read
permissions. Again I got:
2011-10-04 19:41:59 info SASL: Authentication failed for
guest@QPID:SASL(-13): user not found: Password verification failed
I'd be really grateful if someone who knows about this stuff could
suggest what I've done wrong. I can't see why I should be getting "user
not found" with a fairly vanilla set up.
MTIA
Frase
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to set authentication to only authenticate consumers?
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Oh I forgot to say I'm using the C++ broker and a mixture of C++, Java
and Perl clients.
Fraser Adams wrote:
> Hi,
> I haven't done any work playing with authentication, so I'm curious -
> Is it possible to set authentication to only authenticate consumers so
> producers can connect in without needing authentication?
>
> Also is there a good tutorial for getting started with authentication
> - preferable something that starts with the basics to help a total
> authentication noob get something up and running quickly.
>
> My personal view was that I wanted to run our system in a "trust and
> verify" model where we'd audit connections, but some folks in my
> organisation are getting a bit twitchy about that, so I want to keep
> my options open. It's unfortunate as my system is sitting behind a
> firewall on a trusted network and I wanted to have a model that
> maximises business agility by allowing consumers to quickly subscribe
> to the data they need when they need it and do cool stuff with it.
>
> One of my biggest concerns about going down an authentication path is
> the administrative overhead of setting up and managing
> usernames/passwords. How do I do it so that it's not burdonsome to
> allow a new connection - especially if someone needs one in a hurry
> "out of hours". I guess the simple answer might be to have a single
> qpid-subscriber "account", but surely one account/password is little
> better than no authentication at all as anyone who knows this could
> easily set up another consumer client and subscribe to different data.
>
> MTIA
> Frase
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to set authentication to only authenticate consumers?
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Gordon Sim wrote:
> On 10/07/2011 05:50 PM, Fraser Adams wrote:
>> What I'd quite like to be able to do is to log, but not deny if a queue
>> is created that's not one of a named set. I'm suspecting that I can't do
>> that with acl and I might have to write a QMF client to do that.
>
> I think you are right, QMF may be the simplest way to go. You can
> quite easily get events sent out for queue creation and could then
> check the name against an expected set and log any deviation.
Yeah, I suspect that it ought to be pretty easy to do this using
qpid-printevents as a starting point. I've just had a quick look and I
suspect that the only slight gotcha is to cope with the case where the
broker gets restarted and a connection is made straight after. In this
case qpid-printevents was only showing its own bind events. I suspect if
I intercept the bind to the qpid.management exchange and use that to
trigger a getObjects() for all queues/bindings/exchanges to cover that
edge case.
Thanks for all your help yet again!!
Frase
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to set authentication to only authenticate consumers?
Posted by Gordon Sim <gs...@redhat.com>.
On 10/07/2011 05:50 PM, Fraser Adams wrote:
> What I'd quite like to be able to do is to log, but not deny if a queue
> is created that's not one of a named set. I'm suspecting that I can't do
> that with acl and I might have to write a QMF client to do that.
I think you are right, QMF may be the simplest way to go. You can quite
easily get events sent out for queue creation and could then check the
name against an expected set and log any deviation.
> Incidentally, is it possible to get the broker to re-read an acl. I've
> been restarting the broker, but that's not ideal in a live environment.
Yes, there is a QMF method on the broker for re-reading the ACL.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to set authentication to only authenticate consumers?
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
>
> That seems strange to me. For me, if DIGEST-MD5, PLAIN and ANONYMOUS
> are all available, ANONYMOUS is picked by default unless a username is
> set. Are you sure you aren't setting a username?
Pretty certain. As I said earlier it's a pretty basic client that has
string broker = "localhost:5672";
string connectionOptions = "{reconnect: true}";
When I looked at the broker trace it was talking about fadams@QPID,
fadams is the name of the account that I'm using to run the client, but
I've never explicitly used fadams anywhere as a qpid username so
*something* is picking the account name.
>
> I wonder if your sasl lib behaves differently to mine...
Possibly, I'm running Ubuntu - perhaps it's got some subtly different
options. I guess it's no big deal now as I seem to have got things
working generally.
I still think anything to do with security is voodoo magic though, it's
some sort of miracle that I've made it this far :-D
So now I've got another slightly off the wall question :-) So I've got
an acl set up whereby I can have anonymous@QPID to only have publish
rights and named users to subscribe.
What I'd quite like to be able to do is to log, but not deny if a queue
is created that's not one of a named set. I'm suspecting that I can't do
that with acl and I might have to write a QMF client to do that.
Incidentally, is it possible to get the broker to re-read an acl. I've
been restarting the broker, but that's not ideal in a live environment.
Frase
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to set authentication to only authenticate consumers?
Posted by Gordon Sim <gs...@redhat.com>.
On 10/07/2011 01:59 PM, Fraser Adams wrote:
>>
>>> I've just run up a basic C++ client and that asks for a password. It
>>> appears to be sending the account name as the username (in other words
>>> in my case it's saying Authentication failed for fadams@QPID:SASL(-13):
>>> authentication failure: client response doesn't match what we
>>> generated).
>>
>> Hmm, the c++ client doesn't default the username anymore. That was
>> changed some time ago
>> (http://svn.apache.org/viewvc?view=revision&revision=732691), are you
>> sure you are using 0.10 client libs?
>
> I'm fairly sure I am using the 0.10 client libs, when I upgraded from
> 0.8 I did a full make uninstall of 0.8 then a full make/make install of
> 0.10, you might recall I had problems when I initially tried compiling
> 0.10 because the make was picking up the 0.8 libs so I had to do a full
> uninstall of 0.8 before I could make 0.10.
>
> Is there a simple way that I can double check the installed client libs
> are the right version?
I can't think of anything that could be described as simple. The change
I referenced should have been in 0.8 anyway though I think.
> Like I say what I appear to be seeing is that if I don't explicitly set
> mech_list: anonymous plain in qpidd.conf it appears to be trying to
> authenticate via DIGEST-MD5 using the account name as username, when I
> do set the mech_list explicitly it seems to happily send anonymous
That seems strange to me. For me, if DIGEST-MD5, PLAIN and ANONYMOUS are
all available, ANONYMOUS is picked by default unless a username is set.
Are you sure you aren't setting a username?
I wonder if your sasl lib behaves differently to mine...
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to set authentication to only authenticate consumers?
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
>
>> I've just run up a basic C++ client and that asks for a password. It
>> appears to be sending the account name as the username (in other words
>> in my case it's saying Authentication failed for fadams@QPID:SASL(-13):
>> authentication failure: client response doesn't match what we
>> generated).
>
> Hmm, the c++ client doesn't default the username anymore. That was
> changed some time ago
> (http://svn.apache.org/viewvc?view=revision&revision=732691), are you
> sure you are using 0.10 client libs?
I'm fairly sure I am using the 0.10 client libs, when I upgraded from
0.8 I did a full make uninstall of 0.8 then a full make/make install of
0.10, you might recall I had problems when I initially tried compiling
0.10 because the make was picking up the 0.8 libs so I had to do a full
uninstall of 0.8 before I could make 0.10.
Is there a simple way that I can double check the installed client libs
are the right version?
Like I say what I appear to be seeing is that if I don't explicitly set
mech_list: anonymous plain in qpidd.conf it appears to be trying to
authenticate via DIGEST-MD5 using the account name as username, when I
do set the mech_list explicitly it seems to happily send anonymous
Frase
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to set authentication to only authenticate consumers?
Posted by Gordon Sim <gs...@redhat.com>.
On 10/07/2011 12:09 PM, Fraser Adams wrote:
> How would I go about enabling anonymous authentication?
It should be enabled by default.
> I've
> successfully authenticated my basic Java client using the "guest/guest"
> username/password - I'm guessing that's not "anonymous" though as it
> clearly has a name :-).
In the Java client you need to explicitly set the SASL mechanism to use
(else PLAIN is the default). This is to be changed in the next release I
believe, to make the default choose from any of the mutually supported
options.
> I've just run up a basic C++ client and that asks for a password. It
> appears to be sending the account name as the username (in other words
> in my case it's saying Authentication failed for fadams@QPID:SASL(-13):
> authentication failure: client response doesn't match what we generated).
Hmm, the c++ client doesn't default the username anymore. That was
changed some time ago
(http://svn.apache.org/viewvc?view=revision&revision=732691), are you
sure you are using 0.10 client libs?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to set authentication to only authenticate consumers?
Posted by Gordon Sim <gs...@redhat.com>.
On 10/07/2011 12:50 PM, Pavel Moravec wrote:
> qpidd chooses from the available mechanisms the most secure (from available) every time, when possible. I think SASL library itself does not have a priority list itself, the decision is made by qpidd only.
Actually I think it is the other way around. There is no logic in qpidd
(or the cyrus-sasl based clients) to prioritise mechanisms. The user can
configure what the client and broker will support, but its the
underlying sasl library that picks one if there is more than one
mutually acceptable option. The choice is I think based on the perceived
security provided and certainly isn't controlled by the order used in
the mech_list.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to set authentication to only authenticate consumers?
Posted by Pavel Moravec <pm...@redhat.com>.
Hi Frase,
qpidd chooses from the available mechanisms the most secure (from available) every time, when possible. I think SASL library itself does not have a priority list itself, the decision is made by qpidd only.
Kind regards,
Pavel
----- Original Message -----
> From: "Fraser Adams" <fr...@blueyonder.co.uk>
> To: users@qpid.apache.org
> Sent: Friday, October 7, 2011 1:32:02 PM
> Subject: Re: Is it possible to set authentication to only authenticate consumers?
>
> I think I'm sorted now.
>
> I added:
>
> mech_list: anonymous plain
>
> to my qpidd.conf and that seems to work.
>
> out of curiosity does sasl choose the mechanisms in order here?
> Without
> specifying mech_list the broker trace indicated that it was
> supporting a
> wide range of mechanisms including anonymous yet it chose MD5-DIGEST
> (I
> think) when it was initially failing with my c++ client
>
> Frase
>
>
>
> Fraser Adams wrote:
> > Gordon Sim wrote:
> >> On 10/03/2011 06:42 PM, Fraser Adams wrote:
> >>> Is it possible to set authentication to only authenticate
> >>> consumers so
> >>> producers can connect in without needing authentication?
> >>
> >> You can allow both anonymous- and known- users to connect, and
> >> then
> >> use ACLs to only allow the known users to consume while allowing
> >> everyone (including anonymous users) to publish.
> >>
> > Hi Gordon,
> > How would I go about enabling anonymous authentication? I've
> > successfully authenticated my basic Java client using the
> > "guest/guest" username/password - I'm guessing that's not
> > "anonymous"
> > though as it clearly has a name :-).
> >
> > I've just run up a basic C++ client and that asks for a password.
> > It
> > appears to be sending the account name as the username (in other
> > words
> > in my case it's saying Authentication failed for
> > fadams@QPID:SASL(-13): authentication failure: client response
> > doesn't
> > match what we generated).
> >
> > My client is pretty basic and has
> > string broker = "localhost:5672";
> > string connectionOptions = "{reconnect: true}";
> >
> > Now I think that I can add username/password to the connection
> > options
> > and I noticed a |sasl_mechanisms |connection option so I may be
> > able
> > to explicitly set that to anonymous
> >
> > But both of these would require code changes. That's fine in my
> > case
> > here where I can change the code, but in a real world scenario I've
> > got a lot of producers (and I'm not convinced that the developers
> > have
> > necessarily made the connection options configurable) currently
> > connecting to a broker with authentication disabled. I'd like to be
> > able to "authenticate" without them having to change and to add ACL
> > rules to only allow them to produce.
> >
> > I'd have thought that anonymous would have been something that I
> > could
> > enable on the broker config.
> >
> > Have I missed something?
> >
> > MTIA
> > Frase
> >
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > Apache Qpid - AMQP Messaging Implementation
> > Project: http://qpid.apache.org
> > Use/Interact: mailto:users-subscribe@qpid.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to set authentication to only authenticate consumers?
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
I think I'm sorted now.
I added:
mech_list: anonymous plain
to my qpidd.conf and that seems to work.
out of curiosity does sasl choose the mechanisms in order here? Without
specifying mech_list the broker trace indicated that it was supporting a
wide range of mechanisms including anonymous yet it chose MD5-DIGEST (I
think) when it was initially failing with my c++ client
Frase
Fraser Adams wrote:
> Gordon Sim wrote:
>> On 10/03/2011 06:42 PM, Fraser Adams wrote:
>>> Is it possible to set authentication to only authenticate consumers so
>>> producers can connect in without needing authentication?
>>
>> You can allow both anonymous- and known- users to connect, and then
>> use ACLs to only allow the known users to consume while allowing
>> everyone (including anonymous users) to publish.
>>
> Hi Gordon,
> How would I go about enabling anonymous authentication? I've
> successfully authenticated my basic Java client using the
> "guest/guest" username/password - I'm guessing that's not "anonymous"
> though as it clearly has a name :-).
>
> I've just run up a basic C++ client and that asks for a password. It
> appears to be sending the account name as the username (in other words
> in my case it's saying Authentication failed for
> fadams@QPID:SASL(-13): authentication failure: client response doesn't
> match what we generated).
>
> My client is pretty basic and has
> string broker = "localhost:5672";
> string connectionOptions = "{reconnect: true}";
>
> Now I think that I can add username/password to the connection options
> and I noticed a |sasl_mechanisms |connection option so I may be able
> to explicitly set that to anonymous
>
> But both of these would require code changes. That's fine in my case
> here where I can change the code, but in a real world scenario I've
> got a lot of producers (and I'm not convinced that the developers have
> necessarily made the connection options configurable) currently
> connecting to a broker with authentication disabled. I'd like to be
> able to "authenticate" without them having to change and to add ACL
> rules to only allow them to produce.
>
> I'd have thought that anonymous would have been something that I could
> enable on the broker config.
>
> Have I missed something?
>
> MTIA
> Frase
>
>
>
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to set authentication to only authenticate consumers?
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Gordon Sim wrote:
> On 10/03/2011 06:42 PM, Fraser Adams wrote:
>> Is it possible to set authentication to only authenticate consumers so
>> producers can connect in without needing authentication?
>
> You can allow both anonymous- and known- users to connect, and then
> use ACLs to only allow the known users to consume while allowing
> everyone (including anonymous users) to publish.
>
Hi Gordon,
How would I go about enabling anonymous authentication? I've
successfully authenticated my basic Java client using the "guest/guest"
username/password - I'm guessing that's not "anonymous" though as it
clearly has a name :-).
I've just run up a basic C++ client and that asks for a password. It
appears to be sending the account name as the username (in other words
in my case it's saying Authentication failed for fadams@QPID:SASL(-13):
authentication failure: client response doesn't match what we generated).
My client is pretty basic and has
string broker = "localhost:5672";
string connectionOptions = "{reconnect: true}";
Now I think that I can add username/password to the connection options
and I noticed a |sasl_mechanisms |connection option so I may be able to
explicitly set that to anonymous
But both of these would require code changes. That's fine in my case
here where I can change the code, but in a real world scenario I've got
a lot of producers (and I'm not convinced that the developers have
necessarily made the connection options configurable) currently
connecting to a broker with authentication disabled. I'd like to be able
to "authenticate" without them having to change and to add ACL rules to
only allow them to produce.
I'd have thought that anonymous would have been something that I could
enable on the broker config.
Have I missed something?
MTIA
Frase
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to set authentication to only authenticate consumers?
Posted by Gordon Sim <gs...@redhat.com>.
On 10/03/2011 06:42 PM, Fraser Adams wrote:
> Is it possible to set authentication to only authenticate consumers so
> producers can connect in without needing authentication?
You can allow both anonymous- and known- users to connect, and then use
ACLs to only allow the known users to consume while allowing everyone
(including anonymous users) to publish.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Is it possible to set authentication to only authenticate consumers?
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Hi,
I haven't done any work playing with authentication, so I'm curious - Is
it possible to set authentication to only authenticate consumers so
producers can connect in without needing authentication?
Also is there a good tutorial for getting started with authentication -
preferable something that starts with the basics to help a total
authentication noob get something up and running quickly.
My personal view was that I wanted to run our system in a "trust and
verify" model where we'd audit connections, but some folks in my
organisation are getting a bit twitchy about that, so I want to keep my
options open. It's unfortunate as my system is sitting behind a firewall
on a trusted network and I wanted to have a model that maximises
business agility by allowing consumers to quickly subscribe to the data
they need when they need it and do cool stuff with it.
One of my biggest concerns about going down an authentication path is
the administrative overhead of setting up and managing
usernames/passwords. How do I do it so that it's not burdonsome to allow
a new connection - especially if someone needs one in a hurry "out of
hours". I guess the simple answer might be to have a single
qpid-subscriber "account", but surely one account/password is little
better than no authentication at all as anyone who knows this could
easily set up another consumer client and subscribe to different data.
MTIA
Frase
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Significant Headers Exchange interoperability issues noticed
- was Re: JMS client skips string message properties
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Hello again folks.
OK, so having managed to "repair" the Address created by the slightly
broken AddressParser for C++ (see below) I thought that I'd try the same
with Perl, but no!! Despite my Pidjin Perl I *nearly* got there, only to
be thwarted at the last hurdle!!!
It appears that the SWIG Perl binding hides the Variant data type pretty
much completely. I spent most of Sunday tearing my hair out trying
various things to encode the argument value, but I think it's pretty
much impossible to do it in Perl given the existing SWIG binding (if
anyone can figure it out I'd love to know!!!!).
Not one for admitting defeat :-) I took a crash course in SWIG courtesy
of Mr Google.
I've attached for your delectation/humour a C++ binding file to repair
the address and to allow setting properties as UTF8 plus the SWIG
interface file, a Makefile and demo Perl producer/consumer clients.
This stuff allows C++, Perl and Java all to interoperate using the
headers exchange by properly using UTF8 strings as the Address binding
arguments and Message properties.
I hope this stuff is useful - I'm on a mission to promote
interoperability :-D
BTW I know that the right approach is to actually fix the AddressParser,
but I wanted to start with a fix in "user space" to allow C++/Perl
clients to get working without relying on a patch to the Qpid client
runtime code.
Cheers,
Frase
Fraser Adams wrote:
>
>> I agree, this is a bug. I've raised a JIRA and will get that fixed:
>> https://issues.apache.org/jira/browse/QPID-3492
> Hi all,
> For everyone who's been following this thread and have been bitten by
> this issue, in lieu of a patch to the AddressParser itself the
> following code just might help you out. It's a basic "client side"
> code block that, in essence, "repairs" an Address created by the
> broken AddressParser by working its way through the various Address
> blocks until it gets to the x-bindings arguments block it then
> iterates through each argument explicitly setting the encoding of
> VAR_STRING valued arguments to utf8.
>
> My C++ is somewhat rusty so there might well be more elegant/succinct
> ways to find the arguments block, feel free to post back something
> nicer, but the code below seems a fair starter for ten. I've tried it
> with a basic Java producer and the "repaired" bindings match nicely.
>
> Hope this is some use,
> Frase.
>
>
> /*
> // Example Usage:
> string address = "testqueue; {create: receiver, node: {x-declare:
> {arguments: {'qpid.policy_type': ring, 'qpid.max_size': 500000000}},
> x-bindings: [{exchange: 'amq.match', queue: 'testqueue', key: 'data1',
> arguments: {x-match: all, data-service: amqp-delivery, item-owner:
> fadams}}]}}";
>
> Address addr(address);
> addr = utf8EncodeAddress(addr);
>
> Connection connection(broker, connectionOptions);
> try {
> connection.open();
> Session session = connection.createSession();
> Receiver receiver = session.createReceiver(addr);
> receiver.setCapacity(100); // Enable receiver prefetch
>
> ......etc.....
>
> */
>
>
> /*
> There is a bug in the Qpid C++ AddressParser code up to at least
> version 0.12 whereby strings values get encoded
> as raw binary values as opposed to UTF8. In many circumstances this
> doesn't cause significant problems, however
> for the case of bindings to the headers exchange it has the
> potential to create a serious interoperability problem
> as Java producers in particular will certainly set header strings
> as UTF8 Java Strings.
>
> These methods "repair" an Address by re-encoding x-bindings
> argument values as UTF8 strings so that they work
> in an interoperable way with the Headers exchange. Note well the
> use of references as we're changing the
> underlying Address passed to the method.
> */
>
>
> /*
> This method looks for the "x-bindings" Map within a "node" or
> "link" block and if one is found it then
> iterates through the bindings. For each binding that is found the
> "arguments" Map is looked up and if
> an "arguments" Map is found the method iterates though each
> argument explicitly setting the encoding
> of VAR_STRING valued arguments to utf8.
> */
> void utf8EncodeBlock(Variant::Map& block) {
> Variant::Map::iterator i = block.find("x-bindings");
> if (i != block.end()) {
> Variant::List& bindings = i->second.asList();
> for (Variant::List::iterator li = bindings.begin(); li !=
> bindings.end(); li++) {
> Variant::Map& binding = li->asMap();
> i = binding.find("arguments");
> if (i != binding.end()) {
> Variant::Map& arguments = i->second.asMap();
> for (i = arguments.begin(); i != arguments.end(); i++) {
> if (i->second.getType() == VAR_STRING) {
> i->second.setEncoding("utf8");
> }
> }
> }
> }
> }
> }
>
> /*
> This method extracts the "node" and "link" Maps from the options
> part of the Address and passes references
> to them to the utf8EncodeBlock method in order to repair the
> contents of the block.
> */
> Address& utf8EncodeAddress(Address& addr) {
> Variant::Map& options = addr.getOptions();
> Variant::Map::iterator i = options.find("node");
> if (i != options.end()) {
> utf8EncodeBlock(i->second.asMap());
> }
>
> i = options.find("link");
> if (i != options.end()) {
> utf8EncodeBlock(i->second.asMap());
> }
> return addr;
> }
>
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
Re: Significant Headers Exchange interoperability issues noticed
- was Re: JMS client skips string message properties
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
> I agree, this is a bug. I've raised a JIRA and will get that fixed:
> https://issues.apache.org/jira/browse/QPID-3492
Hi all,
For everyone who's been following this thread and have been bitten by
this issue, in lieu of a patch to the AddressParser itself the following
code just might help you out. It's a basic "client side" code block
that, in essence, "repairs" an Address created by the broken
AddressParser by working its way through the various Address blocks
until it gets to the x-bindings arguments block it then iterates through
each argument explicitly setting the encoding of VAR_STRING valued
arguments to utf8.
My C++ is somewhat rusty so there might well be more elegant/succinct
ways to find the arguments block, feel free to post back something
nicer, but the code below seems a fair starter for ten. I've tried it
with a basic Java producer and the "repaired" bindings match nicely.
Hope this is some use,
Frase.
/*
// Example Usage:
string address = "testqueue; {create: receiver, node: {x-declare:
{arguments: {'qpid.policy_type': ring, 'qpid.max_size': 500000000}},
x-bindings: [{exchange: 'amq.match', queue: 'testqueue', key: 'data1',
arguments: {x-match: all, data-service: amqp-delivery, item-owner:
fadams}}]}}";
Address addr(address);
addr = utf8EncodeAddress(addr);
Connection connection(broker, connectionOptions);
try {
connection.open();
Session session = connection.createSession();
Receiver receiver = session.createReceiver(addr);
receiver.setCapacity(100); // Enable receiver prefetch
......etc.....
*/
/*
There is a bug in the Qpid C++ AddressParser code up to at least
version 0.12 whereby strings values get encoded
as raw binary values as opposed to UTF8. In many circumstances this
doesn't cause significant problems, however
for the case of bindings to the headers exchange it has the
potential to create a serious interoperability problem
as Java producers in particular will certainly set header strings as
UTF8 Java Strings.
These methods "repair" an Address by re-encoding x-bindings argument
values as UTF8 strings so that they work
in an interoperable way with the Headers exchange. Note well the use
of references as we're changing the
underlying Address passed to the method.
*/
/*
This method looks for the "x-bindings" Map within a "node" or "link"
block and if one is found it then
iterates through the bindings. For each binding that is found the
"arguments" Map is looked up and if
an "arguments" Map is found the method iterates though each argument
explicitly setting the encoding
of VAR_STRING valued arguments to utf8.
*/
void utf8EncodeBlock(Variant::Map& block) {
Variant::Map::iterator i = block.find("x-bindings");
if (i != block.end()) {
Variant::List& bindings = i->second.asList();
for (Variant::List::iterator li = bindings.begin(); li !=
bindings.end(); li++) {
Variant::Map& binding = li->asMap();
i = binding.find("arguments");
if (i != binding.end()) {
Variant::Map& arguments = i->second.asMap();
for (i = arguments.begin(); i != arguments.end(); i++) {
if (i->second.getType() == VAR_STRING) {
i->second.setEncoding("utf8");
}
}
}
}
}
}
/*
This method extracts the "node" and "link" Maps from the options
part of the Address and passes references
to them to the utf8EncodeBlock method in order to repair the
contents of the block.
*/
Address& utf8EncodeAddress(Address& addr) {
Variant::Map& options = addr.getOptions();
Variant::Map::iterator i = options.find("node");
if (i != options.end()) {
utf8EncodeBlock(i->second.asMap());
}
i = options.find("link");
if (i != options.end()) {
utf8EncodeBlock(i->second.asMap());
}
return addr;
}
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Significant Headers Exchange interoperability issues noticed
- was Re: JMS client skips string message properties
Posted by Gordon Sim <gs...@redhat.com>.
On 09/16/2011 07:08 PM, Fraser Adams wrote:
>
> Thanks for your support in this Gordon. I really do hope that there's an
> elegant way to make interoperability the default position. Sorry if I
> seem to be a broken record on this subject :-)
Not at all, taking the time to keep making that point is appreciated
even if we haven't quite kept pace with the feedback!
>
>>> Is there any way that I can force the address string in C++/perl to be
>>> treated as Unicode in
>>> Receiver receiver = session.createReceiver(address);
> I had a look at the jira you raised, so I guess that the answer to the
> above then is no. Looks pretty down in the guts of Address string
> parsing if it's being treated as a binary. I suspect I might be getting
> away with it by the use of qpid::client in my organisation, perhaps I'll
> have to back off my pressuring to move to qpid::messaging for now -
> that's a real shame.
The only way to do this would be to directly manipulate the Address
object and not rely on the parsing of the stringified form at least for
those arguments.
> On the subject of Address string parsing we spoke a little while about
> maximum queue sizes and the miscellany of uint32, int32 etc. (and
> Integer in the Java AddressParser) that have caused it to be difficult
> to consistently have queue sizes larger than between 2GB or 4GB
> depending on language and where queues are created. I know 0.12 has
> fixed a lot of the broker side issues but the issue still exists in the
> Java AddressParser, do you know what the position is for the C++ and
> Python AddressParsers.
In python I don't believe there is an issue as it does not have the
concept of different types of integer based on their size in memory. In
c++ the values will be converted to int64 if possible. That does
restrict the size a little over a uint64, but certainly it is not as big
an issue as using a int/uint32.
We probably need to enhance the syntax to allow specific constructors or
descriptors to be used to control the exact types (e.g. my_value =
uint64(11) or my_value = 11LL etc).
>> Yes, I agree it is vital. I apologise for the frustration. I really
>> appreciate all your contributions on the list and certainly don't want
>> to drive you nuts! :-)
>>
> No need to apologise, life would be boring if everything was easy :-D I
> wish I'd tried more of this sooner, it was looking into Jiri's problem
> that prompted me to check this out.
>
> Thanks again for your support and kind words about my contributions.
> *hopefully* I'll be in a position to make a proper contribution over the
> next few weeks. My Java implementation of the QMF2 API is coming along
> nicely, the core features are done and I'm at the stage of finishing off
> some of the "shinier" features such as Query subscriptions. I'd hoped to
> have it finished by now, but I only get a chance to do development at
> weekends and I've had a load going on over the last few weeks.
Look forward to it!
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Significant Headers Exchange interoperability issues noticed
- was Re: JMS client skips string message properties
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Thanks for your support in this Gordon. I really do hope that there's an
elegant way to make interoperability the default position. Sorry if I
seem to be a broken record on this subject :-)
>> Is there any way that I can force the address string in C++/perl to be
>> treated as Unicode in
>> Receiver receiver = session.createReceiver(address);
I had a look at the jira you raised, so I guess that the answer to the
above then is no. Looks pretty down in the guts of Address string
parsing if it's being treated as a binary. I suspect I might be getting
away with it by the use of qpid::client in my organisation, perhaps I'll
have to back off my pressuring to move to qpid::messaging for now -
that's a real shame.
On the subject of Address string parsing we spoke a little while about
maximum queue sizes and the miscellany of uint32, int32 etc. (and
Integer in the Java AddressParser) that have caused it to be difficult
to consistently have queue sizes larger than between 2GB or 4GB
depending on language and where queues are created. I know 0.12 has
fixed a lot of the broker side issues but the issue still exists in the
Java AddressParser, do you know what the position is for the C++ and
Python AddressParsers.
>
> Yes, I agree it is vital. I apologise for the frustration. I really
> appreciate all your contributions on the list and certainly don't want
> to drive you nuts! :-)
>
No need to apologise, life would be boring if everything was easy :-D I
wish I'd tried more of this sooner, it was looking into Jiri's problem
that prompted me to check this out.
Thanks again for your support and kind words about my contributions.
*hopefully* I'll be in a position to make a proper contribution over the
next few weeks. My Java implementation of the QMF2 API is coming along
nicely, the core features are done and I'm at the stage of finishing off
some of the "shinier" features such as Query subscriptions. I'd hoped to
have it finished by now, but I only get a chance to do development at
weekends and I've had a load going on over the last few weeks.
Cheers,
Frase
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Significant Headers Exchange interoperability issues noticed
- was Re: JMS client skips string message properties
Posted by Gordon Sim <gs...@redhat.com>.
On 09/16/2011 01:56 PM, Fraser Adams wrote:
> Hi All,
> I was playing around looking at the symptoms Jiri described and I
> decided to check out some nagging concerns I've had with the headers
> exchange.
>
> I'm surprised I've not noticed this before, but I've mainly been looking
> at java and haven't focused on interoperability issues as much as I
> perhaps should have.
>
> So I've tried setting up producers and consumers in c++, java and perl
> with roughly equivalent behaviour. I use an address on each as follows:
>
> C++:
> string address = "test; {create: receiver, node: {x-declare: {arguments:
> {'qpid.policy_type': ring, 'qpid.max_size': 500000000}}, x-bindings:
> [{exchange: 'amq.match', queue: 'test', key: 'data1', arguments:
> {x-match: all, data-service: amqp-delivery, item-owner: fadams}}]}}";
>
> Java (via a JNDI file):
> destination.subscribedAddress1 = test; {create: receiver, node:
> {x-declare: {arguments: {'qpid.policy_type': ring, 'qpid.max_size':
> 500000000}}, x-bindings: [{exchange: 'amq.match', queue: 'test', key:
> 'data1', arguments: {x-match: all, data-service: amqp-delivery,
> item-owner: fadams}}]}}
>
> Perl:
> my $address = "test; {create: receiver, node: {x-declare: {arguments:
> {'qpid.policy_type': ring, 'qpid.max_size': 500000000}}, x-bindings:
> [{exchange: 'amq.match', queue: 'test', key: 'data1', arguments:
> {x-match: all, data-service: amqp-delivery, item-owner: fadams}}]}}";
>
> I had producers writing to amq.match and I found:
>
> perl producer -> perl consumer OK
> perl producer -> c++ consumer OK
> perl producer -> java consumer FAILED
>
> c++ producer -> perl consumer OK
> c++ producer -> c++ consumer OK
> c++ producer -> java consumer FAILED
>
> java producer -> perl consumer FAILED
> java producer -> c++ consumer FAILED
> java producer -> java consumer OK
>
> I know the perl one is a bit of a red herring as it's essentially a
> wrapper round qpid::messaging but I included for completeness. My tests
> were carried out using the 0.10 C++ broker using the qpid::messaging API.
>
> So the thread preceding this has discussed the UTF-8 issues with respect
> to variant types but this is perhaps more insidious. I couldn't quite
> figure it out until I looked using a qpid-config I'd patched so I could
> see headers bindings.
>
> For C++ and perl I get:
> bind [data1] => amq.match {u'item-owner': 'fadams', u'data-service':
> 'amqp-delivery', u'x-match': 'all'}
>
> For java I get:
> bind [data1] => amq.match {u'item-owner': u'fadams', u'data-service':
> u'amqp-delivery', u'x-match': u'all'}
>
> So basically for Java the address string pushes Unicode values onto the
> binding, but for C++/perl the address string pushes (I guess) ASCII
> values onto the binding.
>
>
> This is a pretty big concern for me.
I agree, this is a bug. I've raised a JIRA and will get that fixed:
https://issues.apache.org/jira/browse/QPID-3492
> But one thing is even weirder; in
> my organisation we've got a mixture of c++ and java producers and
> consumers using the headers exchange ,but so far (touch wood) things
> *seem* to be working. However as it happens the C++ producers and
> consumers are using the qpid::client API, so are likely to be binding at
> a lower level rather than using address strings.
>
> I'm trying to persuade the developers to switch to qpid::messaging but
> now I'm worried this could cause interoperability issues.
>
> I have to confess that this is really pretty frustrating. I seem to keep
> getting bitten by string incompatibilities, which I can mostly mitigate
> at the client code level, but problems at the exchange are more painful.
>
>
> I keep on arguing that property strings should default to Unicode to
> ensure interoperability and this convinces me more. Nobody seems to be
> biting on that though - is it just me who's been driven nuts by this??
I agree in principle, just haven't had the time to figure out a change I
liked. I'll have another think about the options...
> Is there any way that I can force the address string in C++/perl to be
> treated as Unicode in
> Receiver receiver = session.createReceiver(address);
>
> Interoperability is really important to me.
Yes, I agree it is vital. I apologise for the frustration. I really
appreciate all your contributions on the list and certainly don't want
to drive you nuts! :-)
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org