You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Fraser Adams <fr...@blueyonder.co.uk> on 2013/03/09 16:01:11 UTC
Java broker exchange questions
Hello all,
I'm somewhat unfamiliar with the Java broker so it'd be great if anyone
knows the answers to these.
1) Does the Java broker support Sequence Number and Initial Value
Exchange. These would be added to the C++ broker using the properties
"qpid.msg_sequence=1" and "qpid.ive=1". I'm pretty sure that the answer
is no, because _virtualHost.getExchangeFactory().createExchange() i.e.
the DefaultExchangeFactory doesn't seem to support passing any extra
properties, but I thought I'd check.
2) Is the xml exchange supported on the Java Broker? I think not because
I'm getting "org.apache.qpid.AMQUnknownExchangeType: Unknown exchange
type: xml" propagated when I try to call createExchange() with type =
"xml", but perhaps there's some a priori config that needs to be put in
place, though looking in
"<qpid-root>/java/broker/src/main/java/org/apache/qpid/server/exchange"
there doesn't seem to be an XMLExchange class, so perhaps it doesn't
exist for the Java broker??
MTIA
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Java broker exchange questions
Posted by Rob Godfrey <ro...@gmail.com>.
On 10 March 2013 09:26, Fraser Adams <fr...@blueyonder.co.uk> wrote:
> On 09/03/13 16:46, Rob Godfrey wrote:
>>
>> Yeah... The biggest gap is probable going to be broker <-> broker
>> federation links. That code did exist previously in the Java Broker,
>> but it disappeared when we ripped out the QMF1 implementation.
>
> I'm curious here, are you saying that federation existed in the Java Broker?
> As far as I was aware federation was only possible between C++ brokers and I
> thought that it was necessary to do some sort of client side bridge to link
> between C++ and Java Brokers or to connect Java Brokers. I was fairly sure
> I'd seen postings to that effect on the user list. Though as I say I'm
> vastly unfamiliar with the Java Broker and this work I'm doing is my first
> journey there.
>
Yeah... there was an implementation of the federation code which
worked with the QMF1 implementation. I'm not sure it was 100%
complete but it covered most of the functionality I was aware of.
Unfortunately the QMF1 stuff kept getting broken and there was no
other way of configuring it (I originally only implemented QMF1 so
that I could add federation to the Java Broker and have a single set
of management tools across the two brokers... it just ended up being
more trouble than it was worth). I can't remember when we ripped out
the QMF1 implementation (0.18 perhaps, but it wasn't *too* long ago).
> Looking at the 0.20 code I've got installed there is a federation directory
> in
>
> <qpid-root>/java/broker/src/main/java/org/apache/qpid/server
>
> but it's empty in 0.20.
>
>
>> We
>> could probably add it back in but it would have to fit within the new
>> internal model for configuration that Alex/Robbie have been working
>> on.
>
> Just to be clear in my mind and to make sure that I'm using the right stuff
> in my QMF mapping by the new internal model are you talking about the stuff
> that lives in:
>
> <qpid-root>/java/broker/src/main/java/org/apache/qpid/server/model
>
> Please say yes :-)
>
Yes :-)
> My QmfManagementAgent is built following the same general patterns as the
> management-http and management-jmx stuff in broker-plugins using the
> ManagementFactory and ManagementPlugin stuff. I've basically used the
> ConfigurationChangeListener interface methods to track changes to the
> server/model ConfiguredObjects and when childAdded is called I use that to
> create concrete QmfAgentData objects that wrap the equivalent
> ConfiguredObject instance. There's *largely* a one-to-one between those
> objects and their QMF equivalents though some of the properties/statistics
> need to be mapped.
>
> To be honest the hardest stuff I've had to cope with to date was due to the
> fact that qpid-config is a bit weird - it doesn't use object references
> correctly and has a hard coded ObjectId for the broker agent for method
> invocation (rather than discovering the broker object's ObjectId) so I had
> to make sure that my Agent copes with that.
>
>
>>
>> Other than that I don't think there are *huge* gaps, just a number of
>> things done annoyingly differently (Gordon and I have tried to rectify
>> this in fits and starts over the years). The QMF schema also tended
>> to make some assumptions based on how the C++ broker was modelled that
>> just didn't hold true for the Java Broker.
>
> If I'm doing it right from what I've seen the differences aren't so very bad
> with the new model. It's only taken me a couple of weekends to get to the
> state where both my QMF GUI and qpid-config can do useful things with the
> Java Broker.
>
> One of the biggest quirks is actually the Virtual Host stuff in the Java
> Broker. Although there are Virtual Host QMF Objects I'm not aware that the
> C++ Broker is really Virtual Host aware? I don't believe that things like
> qpid-config are either - are they?
Yeah - this is another biggie - the C++ broker doesn't really do
vhosts. There are operatins on the QMF "broker" that actually relate
to the vhost on the Java, but there are some things that truely are
"broker" properties.
>
> What I've done in order to allow things like qpid-config to work is to
> prefix queues/exchanges relating to the non-default vhost in other words:
>
> [vhost:<vhost-name>/]<exchange-name> and [vhost:<vhost-name>/]<queue-name>
>
> e.g.
>
> qpid-config exchanges
> Type Exchange Name Attributes
> ============================================================
> fanout vhost:development/amq.fanout --durable
> topic vhost:localhost/amq.topic --durable
> direct vhost:localhost/amq.direct --durable
> fanout vhost:localhost/amq.fanout --durable
> direct vhost:localhost/test.direct --durable
> headers vhost:localhost/amq.match --durable
> headers vhost:development/amq.match --durable
> topic monkey1
> fanout monkey3
> direct vhost:development/ --durable
> direct qmf.default.direct --durable
>
> I need to look to ordering the getObjects() query response base on name to
> make this look neater, but it seems to be the most interoperable approach at
> the moment.
>
>
>> So, personally I'm mostly looking at AMQP 1.0 these days where there
>> isn't the Exchange/Queue division and selection/distribution is a
>> property of the links.
>
> Is there a good simple reference explaining this stuff (aside from poring
> over the AMQP 1.0 specs, which I've not done yet - or could you point me at
> the relevant subsection?). I'm a little nervous 'cause to date I've mostly
> been thinking around exchanges, queues and bindings between them. From what
> I'd understood qpid::messaging was intended to provide an abstraction that
> was agnostic to the AMQP version so I'm assuming that existing
> AddressStrings still ought to work with AMQP 1.0??
>
I'm not sure whether there's any really good summary documentation out
there. In essence AMQP 1.0 only defines the semantics of reliable
message transfer between peers - it does very little in terms of
definition of the internal broker model. It talks of sending messages
between sources and targets and defines some associated semantics
(like the existence of fiters/selectors)... but nothing about what a
broker might do in terms of internal routing.
AddressStrings (in the qpid::messaging sense) should be a purely
client side implementation detail, so if correctly implemented they
should still "work" with AMQP 1.0... but they aren't really something
an AMQP 1.0 broker would be aware of.
>
>> As such I'm not sure I'd personally spend a
>> lot of time writing an XML exchange... but given the XML tooling
>> available in Java then I wouldn't imagine it would be a huge task.
>
> That's why I was curious about the XML exchange. I personally prefer the
> idea of relatively simple subscriptions with finer grained filtering using
> Message Selectors but that was never really an option with C++ broker and
> clients. You could do it with C++ broker and Java clients but the Message
> Selectors were implemented in the client runtime so it wasn't a very
> efficient approach. Andrew Stitcher has just announced initial Message
> Selector support on the C++ broker, so I'm very excited about that.
>
Yeah - I've not looked at Andrew's work in detail... other than to
note that as ever it is close but not identical to the Java
implementation ... I believe that the implementation is only as a
filter on the consumer from a queue... In the Java Broker we also
added the ability to add selectors to a binding between a queue and an
exchange (in fact only on the topic exchange, but this is something we
should extend to all exchanges). Adding selectors to bindings is
necessary if you want to do pub/sub with selectors, else your
temporary subscription queues fill up with stuff that you are never
going to consume.
>> The larger task I would like to undertake is to make it possible to
>> just drop in jars containing new Exchange / Queue types in a pluggable
>> manner.
>
> That sounds really cool.
>
>>
>> Speaking of AMQP 1.0, we're currently working on standardising
>> Management over AMQP 1.0. I'd expect the first draft of this to come
>> out in the next month or so... covering the mechanisms but not
>> attempting to define any "schema" for manageable objects. This is
>> probably where I'll be looking to expend any time I have relative to
>> management work (especially when coupled with AMQP 1.0 over WebSockets
>> and a pure JavaScript AMQP 1.0 implementation).
>
> Rob as you might have gathered given all the work I've done on QMF and the
> QMF GUI I'm *extremely* interested in anything that is afoot with respect to
> Management. I've put a huge amount of effort into that stuff so I'm clearly
> keen to make sure that changes don't break things (well too badly :-)) and
> to evolve things where possible. I guess that the same is true of the broker
> internals such as the
> <qpid-root>/java/broker/src/main/java/org/apache/qpid/server/model which
> seems to be fairly recent.
>
> I'd be really grateful if you could keep me in the loop on the current
> thinking on this.
>
>
Sure - we (the OASIS AMQP Technical Committee) have not published
anything yet, but should do in a couple of weeks. I don't suppose you
work for a firm that is already an OASIS member, do you?
> Re "AMQP 1.0 over WebSockets and a pure JavaScript AMQP 1.0 implementation"
> that sounds pretty cool. How far down the road has this travelled? Would the
> approach be to go from AMQP 1.0 in a browser say via a WebSocket Server such
> as Node.js then via TCP sockets to a Broker.
Adding websockets as a transport to an existing broker is no big deal
- I knocked up a prototype for the Java Broker in about an hour using
Jetty. I'm not overly familiar with library availability in the C/C++
world but I'd expect it should be similarly easy to integrate
websockets as a transport alternative to TCP for other languages too.
> I guess a pure JS Broker
> deployed into Node would be cool too. Is this likely to be limited to
> Node.js or would you expect the WebSocket bridge to be deployable on other
> Servers?
>
> Cheers,
>
> Frase
>
-- Rob
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Java broker exchange questions
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 09/03/13 16:46, Rob Godfrey wrote:
> Yeah... The biggest gap is probable going to be broker <-> broker
> federation links. That code did exist previously in the Java Broker,
> but it disappeared when we ripped out the QMF1 implementation.
I'm curious here, are you saying that federation existed in the Java
Broker? As far as I was aware federation was only possible between C++
brokers and I thought that it was necessary to do some sort of client
side bridge to link between C++ and Java Brokers or to connect Java
Brokers. I was fairly sure I'd seen postings to that effect on the user
list. Though as I say I'm vastly unfamiliar with the Java Broker and
this work I'm doing is my first journey there.
Looking at the 0.20 code I've got installed there is a federation
directory in
<qpid-root>/java/broker/src/main/java/org/apache/qpid/server
but it's empty in 0.20.
> We
> could probably add it back in but it would have to fit within the new
> internal model for configuration that Alex/Robbie have been working
> on.
Just to be clear in my mind and to make sure that I'm using the right
stuff in my QMF mapping by the new internal model are you talking about
the stuff that lives in:
<qpid-root>/java/broker/src/main/java/org/apache/qpid/server/model
Please say yes :-)
My QmfManagementAgent is built following the same general patterns as
the management-http and management-jmx stuff in broker-plugins using the
ManagementFactory and ManagementPlugin stuff. I've basically used the
ConfigurationChangeListener interface methods to track changes to the
server/model ConfiguredObjects and when childAdded is called I use that
to create concrete QmfAgentData objects that wrap the equivalent
ConfiguredObject instance. There's *largely* a one-to-one between those
objects and their QMF equivalents though some of the
properties/statistics need to be mapped.
To be honest the hardest stuff I've had to cope with to date was due to
the fact that qpid-config is a bit weird - it doesn't use object
references correctly and has a hard coded ObjectId for the broker agent
for method invocation (rather than discovering the broker object's
ObjectId) so I had to make sure that my Agent copes with that.
>
> Other than that I don't think there are *huge* gaps, just a number of
> things done annoyingly differently (Gordon and I have tried to rectify
> this in fits and starts over the years). The QMF schema also tended
> to make some assumptions based on how the C++ broker was modelled that
> just didn't hold true for the Java Broker.
If I'm doing it right from what I've seen the differences aren't so very
bad with the new model. It's only taken me a couple of weekends to get
to the state where both my QMF GUI and qpid-config can do useful things
with the Java Broker.
One of the biggest quirks is actually the Virtual Host stuff in the Java
Broker. Although there are Virtual Host QMF Objects I'm not aware that
the C++ Broker is really Virtual Host aware? I don't believe that things
like qpid-config are either - are they?
What I've done in order to allow things like qpid-config to work is to
prefix queues/exchanges relating to the non-default vhost in other words:
[vhost:<vhost-name>/]<exchange-name> and [vhost:<vhost-name>/]<queue-name>
e.g.
qpid-config exchanges
Type Exchange Name Attributes
============================================================
fanout vhost:development/amq.fanout --durable
topic vhost:localhost/amq.topic --durable
direct vhost:localhost/amq.direct --durable
fanout vhost:localhost/amq.fanout --durable
direct vhost:localhost/test.direct --durable
headers vhost:localhost/amq.match --durable
headers vhost:development/amq.match --durable
topic monkey1
fanout monkey3
direct vhost:development/ --durable
direct qmf.default.direct --durable
I need to look to ordering the getObjects() query response base on name
to make this look neater, but it seems to be the most interoperable
approach at the moment.
> So, personally I'm mostly looking at AMQP 1.0 these days where there
> isn't the Exchange/Queue division and selection/distribution is a
> property of the links.
Is there a good simple reference explaining this stuff (aside from
poring over the AMQP 1.0 specs, which I've not done yet - or could you
point me at the relevant subsection?). I'm a little nervous 'cause to
date I've mostly been thinking around exchanges, queues and bindings
between them. From what I'd understood qpid::messaging was intended to
provide an abstraction that was agnostic to the AMQP version so I'm
assuming that existing AddressStrings still ought to work with AMQP 1.0??
> As such I'm not sure I'd personally spend a
> lot of time writing an XML exchange... but given the XML tooling
> available in Java then I wouldn't imagine it would be a huge task.
That's why I was curious about the XML exchange. I personally prefer the
idea of relatively simple subscriptions with finer grained filtering
using Message Selectors but that was never really an option with C++
broker and clients. You could do it with C++ broker and Java clients but
the Message Selectors were implemented in the client runtime so it
wasn't a very efficient approach. Andrew Stitcher has just announced
initial Message Selector support on the C++ broker, so I'm very excited
about that.
> The larger task I would like to undertake is to make it possible to
> just drop in jars containing new Exchange / Queue types in a pluggable
> manner.
That sounds really cool.
>
> Speaking of AMQP 1.0, we're currently working on standardising
> Management over AMQP 1.0. I'd expect the first draft of this to come
> out in the next month or so... covering the mechanisms but not
> attempting to define any "schema" for manageable objects. This is
> probably where I'll be looking to expend any time I have relative to
> management work (especially when coupled with AMQP 1.0 over WebSockets
> and a pure JavaScript AMQP 1.0 implementation).
Rob as you might have gathered given all the work I've done on QMF and
the QMF GUI I'm *extremely* interested in anything that is afoot with
respect to Management. I've put a huge amount of effort into that stuff
so I'm clearly keen to make sure that changes don't break things (well
too badly :-)) and to evolve things where possible. I guess that the
same is true of the broker internals such as the
<qpid-root>/java/broker/src/main/java/org/apache/qpid/server/model which
seems to be fairly recent.
I'd be really grateful if you could keep me in the loop on the current
thinking on this.
Re "AMQP 1.0 over WebSockets and a pure JavaScript AMQP 1.0
implementation" that sounds pretty cool. How far down the road has this
travelled? Would the approach be to go from AMQP 1.0 in a browser say
via a WebSocket Server such as Node.js then via TCP sockets to a Broker.
I guess a pure JS Broker deployed into Node would be cool too. Is this
likely to be limited to Node.js or would you expect the WebSocket bridge
to be deployable on other Servers?
Cheers,
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Java broker exchange questions
Posted by Robbie Gemmell <ro...@gmail.com>.
Correcting myself, it occurs to me that isn't just on trunk, it actually
got included in 0.20.
Robbie
On 9 March 2013 19:20, Robbie Gemmell <ro...@gmail.com> wrote:
> On 9 March 2013 16:46, Rob Godfrey <ro...@gmail.com> wrote:
> <snip>
>
> The larger task I would like to undertake is to make it possible to
>> just drop in jars containing new Exchange / Queue types in a pluggable
>> manner.
>>
>>
> That is actually already supported on trunk for Exchange types, and is how
> even the base Exchange types from the prior AMQP specs are now loaded (we
> enforce that the base types are present though).
>
> Queues are not loaded in the same way yet as that was definitely a larger
> task calling for more of a rewrite in order to implement.
>
Re: Java broker exchange questions
Posted by Rob Godfrey <ro...@gmail.com>.
On 10 March 2013 12:28, Fraser Adams <fr...@blueyonder.co.uk> wrote:
> On 10/03/13 09:51, Rob Godfrey wrote:
>>
>> On 10 March 2013 09:42, Fraser Adams <fr...@blueyonder.co.uk>
>> wrote:
>>>
>>> I'd really like to understand more about how this is currently working in
>>> AMQP 1.0/Proton do you have some example Addresses that are being used -
>>> I'm
>>> particularly interested in how an existing AddressString containing a
>>> headers exchange and bindings to that might work.
>>
>> So, what the Java Broker does under the covers is actually to create a
>> temporary subscription queue.
>
> That's interesting, that sounds fairly analogous to the federation in the
> C++ broker for things like dynamic exchange route and static exchange routes
> where *logically* exchanges are being linked together, but at the
> implementation level a temporary queue gets created.
>
> I guess from your previous response about what AMQP 1.0 does (and does not
> :-)) specify an awful lot is potentially at the level of "implementation
> detail". That feels like a rich seam of confusion that we're likely to be
> mining for quite some time I suspect :-D
>
There definitely has to be a clearer understanding of the distinction
between the protocol and the capabilities of any given intermediary
(such as a broker).
As an end user I want to be able to "subscribe to a topic". In AMQP
1.0 that means setting up a link to receive messages from a source
which acts as a topic. How to configure that address to behave like a
topic is not part of the core protocol.
The definition of standard node types and how to configure them will
be encouraged through the management definitions... however the 80%
use case where the client applications do not need to reconfigure the
internal wiring of the intermediary is covered by the core protocol.
In some ways the scope of the core protocol can be thought of as very
similar to the scope of the JMS API (albeit at a wire level rather
than an API level).
>
>> One could actually implement
>> subscriptions to exchanges such that published messages are not
>> acknowledged until they have been flowed to all recipients... but that
>> would have been a bigger piece of work.
>
> That's interesting. My gut feeling is that it might be pretty dangerous.
> There's already a lot of scope for memory issues. I've been bitten in the
> past by quirks related to prefetch, so for example messages get sent to the
> prefetch queue on say slow consumers and end up building up on the broker
> because it can't remove the references until the client has acknowledged.
> When one actually thinks about what's going on it all makes sense, but it
> took me an age to get under the skin of this sort of behaviour and I wonder
> if some of these higher level abstractions and implementation details could
> leave people scratching their heads. I had enough problems thinking "my
> messages are hitting a queue with ring policy why on earth is the broker
> memory growing - and growing". If it was only *logically* hitting an
> exchange I suspect that it would *really* have melted my mind (and I've got
> a tenuous enough grip on sanity as it is :->)
>
Yes - though I think by building better abstractions we actually have
a chance to make things less complex. Shoehorning everything into the
exchange->queue model with prefetch made certain scenarios (especially
with regard to topics) really mind-bending to try to understand from
an operational point of view. Looking at the broker you'll just see
hundreds of temporary queues... which "topic" are they subscribing
too... which "topics" are growing... etc...
>
>>
>> Ultimately I want to just add "topic" nodes rather than abusing
>> exchanges for this purpose (I had a 10 line patch that did this that
>> is probably still sitting around somewhere). The primary
>> implementation difference is that messages at the head of a "topic"
>> get silently dropped if there are no subscriptions that have not yet
>> passed that message.
>
> Are you suggesting here that your "topic" node implements an internal queue
> ("messages at the head of a "topic" get silently dropped" is suggesting that
> to me).
>
> I have some issues with that and again it's analogous to federation.
> Ultimately I've been forced into using queue routes for federation due to
> the limitations of the "internal queues" that have been implemented for the
> "exchange routes" ultimately having non-configurable internal queues is a
> pain in the butt - I'd really have liked to be able to specify the size and
> policy of the internal queue.
>
So my "topic" implementation was of a standard queue (which can then
have a policy such as ring queue behaviour layered on top), where
"consumers" act more like browsers (they are non-destructive
consumers). If we think of each consumer as actually just being a
pointer into the queue, then when all active consumers have "past" a
given message on a "topic" then it can be discarded. (There is also a
use case for a topic which discards based on message age and allows
subscribers to join the topic as if it were some past point in time).
> I'm concerned that some of these thoughts around AMQP 1.0 might a) become
> inconsistent between the C++ and Java brokers and b) cause problems like I
> suggest above where the may not be as configurable as users need.
>
Certainly I want to reduce inconsistencies between the brokers where
possible (we're never going to count number of mallocs in the Java
Broker :) ). I'd also hope on the Java side at least to make things
more configurable (and for that configuration to be more easy to
understand). In general I'd like to see all of Qpid aiming to track
the AMQP specifications much more closely as a way to drive
functionality, and drive standardisation where it is not yet present.
>
>>> Perhaps, but it's probably a good topic to start, the AMQP 1.0 work has
>>> been
>>> quietly going on in dev land and I think this is probably a good time to
>>> start sharing the love a bit on the user list.
>>>
>> +1. We definitely need to talk more about AMQP 1.0 on this list as
>> well as better explaining what things are AMQP 1.0 and what things are
>> "Qpid".
>>
> Cool I'm glad that you agree. It's definitely an interesting time but there
> may be some pain ahead for users as things move forward and good
> communication (and documentation) seems to me to be the best way to mitigate
> this.
>
>
Yeah... I think we need to make the transition to 1.0 as easy as
possible for existing users. Things such as the qpid messaging API,
(or using JMS as an API) will make this easier. Overall I think AMQP
1.0 will lead to new and better ways of doing things.
-- Rob
> BTW you asked previously "I don't suppose you work for a firm that is
> already an OASIS member, do you?" unfortunately not. As it happens all of my
> Qpid work is done in my own time at weekends
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Java broker exchange questions
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 10/03/13 09:51, Rob Godfrey wrote:
> On 10 March 2013 09:42, Fraser Adams <fr...@blueyonder.co.uk> wrote:
>> I'd really like to understand more about how this is currently working in
>> AMQP 1.0/Proton do you have some example Addresses that are being used - I'm
>> particularly interested in how an existing AddressString containing a
>> headers exchange and bindings to that might work.
> So, what the Java Broker does under the covers is actually to create a
> temporary subscription queue.
That's interesting, that sounds fairly analogous to the federation in
the C++ broker for things like dynamic exchange route and static
exchange routes where *logically* exchanges are being linked together,
but at the implementation level a temporary queue gets created.
I guess from your previous response about what AMQP 1.0 does (and does
not :-)) specify an awful lot is potentially at the level of
"implementation detail". That feels like a rich seam of confusion that
we're likely to be mining for quite some time I suspect :-D
> One could actually implement
> subscriptions to exchanges such that published messages are not
> acknowledged until they have been flowed to all recipients... but that
> would have been a bigger piece of work.
That's interesting. My gut feeling is that it might be pretty dangerous.
There's already a lot of scope for memory issues. I've been bitten in
the past by quirks related to prefetch, so for example messages get sent
to the prefetch queue on say slow consumers and end up building up on
the broker because it can't remove the references until the client has
acknowledged. When one actually thinks about what's going on it all
makes sense, but it took me an age to get under the skin of this sort of
behaviour and I wonder if some of these higher level abstractions and
implementation details could leave people scratching their heads. I had
enough problems thinking "my messages are hitting a queue with ring
policy why on earth is the broker memory growing - and growing". If it
was only *logically* hitting an exchange I suspect that it would
*really* have melted my mind (and I've got a tenuous enough grip on
sanity as it is :->)
>
> Ultimately I want to just add "topic" nodes rather than abusing
> exchanges for this purpose (I had a 10 line patch that did this that
> is probably still sitting around somewhere). The primary
> implementation difference is that messages at the head of a "topic"
> get silently dropped if there are no subscriptions that have not yet
> passed that message.
Are you suggesting here that your "topic" node implements an internal
queue ("messages at the head of a "topic" get silently dropped" is
suggesting that to me).
I have some issues with that and again it's analogous to federation.
Ultimately I've been forced into using queue routes for federation due
to the limitations of the "internal queues" that have been implemented
for the "exchange routes" ultimately having non-configurable internal
queues is a pain in the butt - I'd really have liked to be able to
specify the size and policy of the internal queue.
I'm concerned that some of these thoughts around AMQP 1.0 might a)
become inconsistent between the C++ and Java brokers and b) cause
problems like I suggest above where the may not be as configurable as
users need.
>> Perhaps, but it's probably a good topic to start, the AMQP 1.0 work has been
>> quietly going on in dev land and I think this is probably a good time to
>> start sharing the love a bit on the user list.
>>
> +1. We definitely need to talk more about AMQP 1.0 on this list as
> well as better explaining what things are AMQP 1.0 and what things are
> "Qpid".
>
Cool I'm glad that you agree. It's definitely an interesting time but
there may be some pain ahead for users as things move forward and good
communication (and documentation) seems to me to be the best way to
mitigate this.
BTW you asked previously "I don't suppose you work for a firm that is
already an OASIS member, do you?" unfortunately not. As it happens all
of my Qpid work is done in my own time at weekends
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Java broker exchange questions
Posted by Rob Godfrey <ro...@gmail.com>.
On 10 March 2013 09:42, Fraser Adams <fr...@blueyonder.co.uk> wrote:
>
>> Yeah - from an AMQP 1.0 point of view one might consider Queues and
>> Exchanges both as types of "Nodes" (the 1.0 transport for the Java
>> Broker actually allows you to publish directly to Queues and subscribe
>> to Exchanges)... how to present this and the 0.x view as a single
>> coherent model will be "interesting" (are bindings links from
>> exchanges to queues I wonder)
>
> As per previous post I'm *really* interested in this going forward as I
> suspect are many people. Publishing to a queue seems fairly "normal" given
> that the default direct exchange provided a mechanism to use a producer
> address that just looked like a queue, but subscribing to an exchange seems
> weirder. In the latter case clearly a topic address made it look like this
> was happening, but a temporary queue was always put in place *actually*
> subscribing to exchanges seems more unusual as one would normally expect
> some sort of "elastic" buffering to decouple producers and consumers. I
> guess that with Qpid there's the concept of prefetching messages so I guess
> that this gives the necessary buffering (unless synchronous behaviour has
> been specified).
>
> I'd really like to understand more about how this is currently working in
> AMQP 1.0/Proton do you have some example Addresses that are being used - I'm
> particularly interested in how an existing AddressString containing a
> headers exchange and bindings to that might work.
So, what the Java Broker does under the covers is actually to create a
temporary subscription queue. One could actually implement
subscriptions to exchanges such that published messages are not
acknowledged until they have been flowed to all recipients... but that
would have been a bigger piece of work.
Ultimately I want to just add "topic" nodes rather than abusing
exchanges for this purpose (I had a 10 line patch that did this that
is probably still sitting around somewhere). The primary
implementation difference is that messages at the head of a "topic"
get silently dropped if there are no subscriptions that have not yet
passed that message.
>
>>
>> -- Rob (going seriously off-topic)
>>
>>
> Perhaps, but it's probably a good topic to start, the AMQP 1.0 work has been
> quietly going on in dev land and I think this is probably a good time to
> start sharing the love a bit on the user list.
>
+1. We definitely need to talk more about AMQP 1.0 on this list as
well as better explaining what things are AMQP 1.0 and what things are
"Qpid".
-- Rob
>
> Frase
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Java broker exchange questions
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
> Yeah - from an AMQP 1.0 point of view one might consider Queues and
> Exchanges both as types of "Nodes" (the 1.0 transport for the Java
> Broker actually allows you to publish directly to Queues and subscribe
> to Exchanges)... how to present this and the 0.x view as a single
> coherent model will be "interesting" (are bindings links from
> exchanges to queues I wonder)
As per previous post I'm *really* interested in this going forward as I
suspect are many people. Publishing to a queue seems fairly "normal"
given that the default direct exchange provided a mechanism to use a
producer address that just looked like a queue, but subscribing to an
exchange seems weirder. In the latter case clearly a topic address made
it look like this was happening, but a temporary queue was always put in
place *actually* subscribing to exchanges seems more unusual as one
would normally expect some sort of "elastic" buffering to decouple
producers and consumers. I guess that with Qpid there's the concept of
prefetching messages so I guess that this gives the necessary buffering
(unless synchronous behaviour has been specified).
I'd really like to understand more about how this is currently working
in AMQP 1.0/Proton do you have some example Addresses that are being
used - I'm particularly interested in how an existing AddressString
containing a headers exchange and bindings to that might work.
>
> -- Rob (going seriously off-topic)
>
>
Perhaps, but it's probably a good topic to start, the AMQP 1.0 work has
been quietly going on in dev land and I think this is probably a good
time to start sharing the love a bit on the user list.
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Java broker exchange questions
Posted by Rob Godfrey <ro...@gmail.com>.
On 9 March 2013 20:20, Robbie Gemmell <ro...@gmail.com> wrote:
> On 9 March 2013 16:46, Rob Godfrey <ro...@gmail.com> wrote:
> <snip>
>
>> The larger task I would like to undertake is to make it possible to
>> just drop in jars containing new Exchange / Queue types in a pluggable
>> manner.
>>
>>
> That is actually already supported on trunk for Exchange types, and is how
> even the base Exchange types from the prior AMQP specs are now loaded (we
> enforce that the base types are present though).
Ah yes - I think you mentioned that to me... but it slipped my mind :)
That is a huge improvement!
>
> Queues are not loaded in the same way yet as that was definitely a larger
> task calling for more of a rewrite in order to implement.
Yeah - from an AMQP 1.0 point of view one might consider Queues and
Exchanges both as types of "Nodes" (the 1.0 transport for the Java
Broker actually allows you to publish directly to Queues and subscribe
to Exchanges)... how to present this and the 0.x view as a single
coherent model will be "interesting" (are bindings links from
exchanges to queues I wonder)
-- Rob (going seriously off-topic)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Java broker exchange questions
Posted by Robbie Gemmell <ro...@gmail.com>.
On 9 March 2013 16:46, Rob Godfrey <ro...@gmail.com> wrote:
<snip>
> The larger task I would like to undertake is to make it possible to
> just drop in jars containing new Exchange / Queue types in a pluggable
> manner.
>
>
That is actually already supported on trunk for Exchange types, and is how
even the base Exchange types from the prior AMQP specs are now loaded (we
enforce that the base types are present though).
Queues are not loaded in the same way yet as that was definitely a larger
task calling for more of a rewrite in order to implement.
Re: Java broker exchange questions
Posted by Rob Godfrey <ro...@gmail.com>.
On 9 March 2013 17:30, Fraser Adams <fr...@blueyonder.co.uk> wrote:
> Cheers Rob,
> That all concurs with my observations, so at least I've not missed anything.
>
> Part of the reason I asked is because of consistency with the C++ broker
> which does support these things. I'm currently most of the way through a
> QmfManagementPlugin which allows the Java Broker to be controlled by things
> like qpid-config and my QMF GUI. Most of the main bits are there now but
> when I test with qpid-config I'm keen to make sure that I map whatever I can
> into the Java Broker's internal model.
Yeah... The biggest gap is probable going to be broker <-> broker
federation links. That code did exist previously in the Java Broker,
but it disappeared when we ripped out the QMF1 implementation. We
could probably add it back in but it would have to fit within the new
internal model for configuration that Alex/Robbie have been working
on.
Other than that I don't think there are *huge* gaps, just a number of
things done annoyingly differently (Gordon and I have tried to rectify
this in fits and starts over the years). The QMF schema also tended
to make some assumptions based on how the C++ broker was modelled that
just didn't hold true for the Java Broker.
>
> TBH I've never used IVE or sequence numbers myself, but I'm pretty sure that
> I have seen postings related to these in the context of the C++ broker so
> I'm guessing that some people do use them, I guess Gordon will probably
> recall such queries better than most.
>
> XML exchange is another matter and although not part of the AMQP standard
> it's got a lot of potential, that said the Java broker supports JMS Message
> Selectors which I guess probably cater for most of the use cases where the
> XML exchange may have been beneficial. Andrew Stitcher has started work on
> Message Selectors for the C++ broker (which is uber cool IMHO, I've been
> hoping for that for ages) so perhaps ultimately fully interoperable Message
> Selectors are a better way forward than XML exchanges?
>
So, personally I'm mostly looking at AMQP 1.0 these days where there
isn't the Exchange/Queue division and selection/distribution is a
property of the links. As such I'm not sure I'd personally spend a
lot of time writing an XML exchange... but given the XML tooling
available in Java then I wouldn't imagine it would be a huge task.
The larger task I would like to undertake is to make it possible to
just drop in jars containing new Exchange / Queue types in a pluggable
manner.
Speaking of AMQP 1.0, we're currently working on standardising
Management over AMQP 1.0. I'd expect the first draft of this to come
out in the next month or so... covering the mechanisms but not
attempting to define any "schema" for manageable objects. This is
probably where I'll be looking to expend any time I have relative to
management work (especially when coupled with AMQP 1.0 over WebSockets
and a pure JavaScript AMQP 1.0 implementation).
Cheers,
Rob
> I'll keep you posted and thanks again for the response.
> Frase
>
>
>
> On 09/03/13 15:37, Rob Godfrey wrote:
>>
>> Hi Fraser
>>
>> On 9 March 2013 16:01, Fraser Adams <fr...@blueyonder.co.uk> wrote:
>>>
>>> Hello all,
>>> I'm somewhat unfamiliar with the Java broker so it'd be great if anyone
>>> knows the answers to these.
>>>
>>> 1) Does the Java broker support Sequence Number and Initial Value
>>> Exchange.
>>> These would be added to the C++ broker using the properties
>>> "qpid.msg_sequence=1" and "qpid.ive=1". I'm pretty sure that the answer
>>> is
>>> no, because _virtualHost.getExchangeFactory().createExchange() i.e. the
>>> DefaultExchangeFactory doesn't seem to support passing any extra
>>> properties,
>>> but I thought I'd check.
>>>
>> Nope - I think someone knocked up some code for this at some point,
>> but it was never checked in. I've never really been sure of the use
>> case for the initial value exchange.
>>
>>> 2) Is the xml exchange supported on the Java Broker? I think not because
>>> I'm
>>> getting "org.apache.qpid.AMQUnknownExchangeType: Unknown exchange type:
>>> xml"
>>> propagated when I try to call createExchange() with type = "xml", but
>>> perhaps there's some a priori config that needs to be put in place,
>>> though
>>> looking in
>>> "<qpid-root>/java/broker/src/main/java/org/apache/qpid/server/exchange"
>>> there doesn't seem to be an XMLExchange class, so perhaps it doesn't
>>> exist
>>> for the Java broker??
>>>
>> Correct - no XMLExchange either. Probably wouldn't be a huge thing to
>> implement one... but to be honest no-one ever asked and it's not part
>> of the AMQP standard.
>>
>> Cheers,
>> Rob
>>
>>> MTIA
>>> Frase
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Java broker exchange questions
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Cheers Rob,
That all concurs with my observations, so at least I've not missed anything.
Part of the reason I asked is because of consistency with the C++ broker
which does support these things. I'm currently most of the way through a
QmfManagementPlugin which allows the Java Broker to be controlled by
things like qpid-config and my QMF GUI. Most of the main bits are there
now but when I test with qpid-config I'm keen to make sure that I map
whatever I can into the Java Broker's internal model.
TBH I've never used IVE or sequence numbers myself, but I'm pretty sure
that I have seen postings related to these in the context of the C++
broker so I'm guessing that some people do use them, I guess Gordon will
probably recall such queries better than most.
XML exchange is another matter and although not part of the AMQP
standard it's got a lot of potential, that said the Java broker supports
JMS Message Selectors which I guess probably cater for most of the use
cases where the XML exchange may have been beneficial. Andrew Stitcher
has started work on Message Selectors for the C++ broker (which is uber
cool IMHO, I've been hoping for that for ages) so perhaps ultimately
fully interoperable Message Selectors are a better way forward than XML
exchanges?
I'll keep you posted and thanks again for the response.
Frase
On 09/03/13 15:37, Rob Godfrey wrote:
> Hi Fraser
>
> On 9 March 2013 16:01, Fraser Adams <fr...@blueyonder.co.uk> wrote:
>> Hello all,
>> I'm somewhat unfamiliar with the Java broker so it'd be great if anyone
>> knows the answers to these.
>>
>> 1) Does the Java broker support Sequence Number and Initial Value Exchange.
>> These would be added to the C++ broker using the properties
>> "qpid.msg_sequence=1" and "qpid.ive=1". I'm pretty sure that the answer is
>> no, because _virtualHost.getExchangeFactory().createExchange() i.e. the
>> DefaultExchangeFactory doesn't seem to support passing any extra properties,
>> but I thought I'd check.
>>
> Nope - I think someone knocked up some code for this at some point,
> but it was never checked in. I've never really been sure of the use
> case for the initial value exchange.
>
>> 2) Is the xml exchange supported on the Java Broker? I think not because I'm
>> getting "org.apache.qpid.AMQUnknownExchangeType: Unknown exchange type: xml"
>> propagated when I try to call createExchange() with type = "xml", but
>> perhaps there's some a priori config that needs to be put in place, though
>> looking in
>> "<qpid-root>/java/broker/src/main/java/org/apache/qpid/server/exchange"
>> there doesn't seem to be an XMLExchange class, so perhaps it doesn't exist
>> for the Java broker??
>>
> Correct - no XMLExchange either. Probably wouldn't be a huge thing to
> implement one... but to be honest no-one ever asked and it's not part
> of the AMQP standard.
>
> Cheers,
> Rob
>
>> MTIA
>> Frase
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Java broker exchange questions
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Fraser
On 9 March 2013 16:01, Fraser Adams <fr...@blueyonder.co.uk> wrote:
> Hello all,
> I'm somewhat unfamiliar with the Java broker so it'd be great if anyone
> knows the answers to these.
>
> 1) Does the Java broker support Sequence Number and Initial Value Exchange.
> These would be added to the C++ broker using the properties
> "qpid.msg_sequence=1" and "qpid.ive=1". I'm pretty sure that the answer is
> no, because _virtualHost.getExchangeFactory().createExchange() i.e. the
> DefaultExchangeFactory doesn't seem to support passing any extra properties,
> but I thought I'd check.
>
Nope - I think someone knocked up some code for this at some point,
but it was never checked in. I've never really been sure of the use
case for the initial value exchange.
> 2) Is the xml exchange supported on the Java Broker? I think not because I'm
> getting "org.apache.qpid.AMQUnknownExchangeType: Unknown exchange type: xml"
> propagated when I try to call createExchange() with type = "xml", but
> perhaps there's some a priori config that needs to be put in place, though
> looking in
> "<qpid-root>/java/broker/src/main/java/org/apache/qpid/server/exchange"
> there doesn't seem to be an XMLExchange class, so perhaps it doesn't exist
> for the Java broker??
>
Correct - no XMLExchange either. Probably wouldn't be a huge thing to
implement one... but to be honest no-one ever asked and it's not part
of the AMQP standard.
Cheers,
Rob
> MTIA
> Frase
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org