Re: Qpid Dispatch Router component

[Rob Godfrey <ro...@gmail.com>]

On 9 October 2013 19:06, Ted Ross <tr...@redhat.com> wrote:

> Rob,
>
> These are good points.  Let me start with management.
>
> I view Dispatch as a bit of a testing ground for the emerging AMQP
> Management specification.  I would claim that at this point, the
> specification is not ready for prime-time.  With regard to the address, the
> specification uses "$management".  Perhaps I'm mistaken, but I took that to
> mean "place-holder for an as-yet unspecified literal string".  Clearly
> there needs to be a way to address multiple distinct container agents
> through the same connection or link.  If this is not the case, then it will
> need to be addressed in the technical committee.
>
>
"$management" is meant as a literal.  When you pair this with global
addressing you can define addresses for each of the management nodes in
separate container instances /<instance A>/$management /<instance
B>/$management, etc... moreover you may wish to have the management node at
each instance aware of the management nodes of instances to which it is
connected, thus by issuing the discover commands to the management node at
the local router you have connected to you can potentially find all
management nodes in the connected network.


> I've personally been tracking both the management and addressing
> specifications circulating through the technical committee and I expect
> that Dispatch will conform to (and in fact prove out) both specifications.
>  I'm not aware of any other project within Qpid that is implementing the
> management specification (perhaps the Java broker is).
>
>
My main concern is that I believe that Qpid should be primarily directed at
implementing AMQP standards, and building resuable toolkits and components
that fit into any AMQP network.  I'd be very concerned if we were inventing
alternative management protocols, or building components that only
interoperated with other Qpid tools.

So, personally I'd like to see statement around components saying that they
will be fully implementing emerging AMQP Management, AMQP addressing, etc.
standards, and that we as a project then ensure we stick to these goals.


> With regard to interaction with other Qpid and Apache projects, Dispatch
> is already proved interoperable with the Proton Messenger and
> qpid::messaging clients.  It should also be usable as interconnect between
> clients and the Qpid brokers via AMQP 1.0 and between multiple brokers as a
> federation interconnect.  Some experimentation has been done using the
> ActiveMQ broker as well.
>
> With regard to protocols, I would be opposed to trying to implement AMQP
> 0-* due to the asymmetric nature of those protocols.  I do think, however,
> that Dispatch could make a very good platform for an edge concentrator for
> lighter protocols like MQTT.
>
>
I guess what I'm driving at is that it would be good to have a proper
scope/charter statement for each of our "components" that we as a project
had broad agreement on which do restrict us from just dropping anything we
feel like into a component.  I'd like to do that for Dispatch before we
elevate it, and I think we should ensure that we also have such statements
for the JMS client (which I think should be easy) and Proton (which may be
a little trickier).

Cheers,
Rob


> Regards,
>
> -Ted
>
>
> On 10/09/2013 12:20 PM, Rob Godfrey wrote:
>
>> Hi Ted,
>>
>> I think before we make this a full sub project, it would be good to have
>> clarity on exactly the proposed scope of Dispatch, how it is expected to
>> interact with other components within Qpid, or within wider AMQP networks.
>> I think in retrospect we didn't do this clearly enough with Proton (for
>> example).
>>
>> Moreover I would personally like to understand which AMQP standards it
>> will
>> be looking to implement, and which not.  For instance I notice this line
>> in
>> the docs for Dispatch:
>>
>> *Address**Description* /_local/agentThe management agent on the attached
>>
>> router/container. This address would be used by an endpoint that is a
>> management client/console/tool wishing to access management data from the
>> attached container.
>> Which doesn't seem to conform with the proposed management specification
>> for AMQP, nor does the document make any mention of how dispatch is to be
>> managed.
>>
>>
>> Cheers,
>> Rob
>>
>>
>> On 9 October 2013 17:22, Ted Ross <tr...@redhat.com> wrote:
>>
>>  The AMQP Router project (Qpid Dispatch, announced previously on the user
>>> list) is gaining in community interest and is nearing the point where a
>>> first release is appropriate. In preparation for a release, I proposethat
>>> this sub-project follow the lead of both Proton and the AMQP1.0 JMS
>>> projects. This involves:
>>>
>>> 1. Moving the code from qpid/extras to
>>>     http://svn.apache.org/repos/****asf/qpid/dispatch<http://svn.apache.org/repos/**asf/qpid/dispatch>
>>> <http://svn.**apache.org/repos/asf/qpid/**dispatch<http://svn.apache.org/repos/asf/qpid/dispatch>
>>> >
>>>
>>> ,
>>> 2. Requesting, by vote, the creation of a JIRA project to track its
>>>     issues and releases.
>>>
>>> Unless there are objections, I will move forward with the above two
>>> tasks.
>>>
>>> Regards,
>>>
>>> -Ted
>>>
>>>
>>>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.**org<de...@qpid.apache.org>
> For additional commands, e-mail: dev-help@qpid.apache.org
>
>

Re: Qpid Dispatch Router component

[Gordon Sim <gs...@redhat.com>]

On 10/10/2013 12:09 PM, Rob Godfrey wrote:
> On 10 October 2013 12:46, Gordon Sim <gs...@redhat.com> wrote:
>> On 10/10/2013 10:58 AM, Rob Godfrey wrote:
>>> I think the point of Qpid (vs. any other messaging
>>> implementation at Apache or elsewhere) is to implement the AMQP
>>> specification.
>>
>> I have no disagreement when the AMQP specification is what is currently
>> published.
>>
>> My concern is where that is defined to be an expanding, all-encompassing
>> effort that ring fences whole areas of behaviour and sets itself up as the
>> only legitimate avenue for exploration.
>
> Do you believe that is what the OASIS TCs are doing?  If so in which areas?

The idea of a general policy that any Qpid component must support any 
emerging standard from those TCs in a particular area and cannot explore 
alternatives or anything that may overlap does feel very much like this.

I am not saying that is what you meant, merely expressing that I would 
not be comfortable with such a policy if it was.

>> Expanding beyond the current specification needs a more diverse source of
>> ideas and a more open, transparent and collaborative process.
>>
> While I agree that a more open process is most definitely desirable, I
> would argue that the OASIS TCs (sadly) currently have a much more diverse
> membership than Qpid committers.Also note that you do not need to be a
> member of OASIS to comment on work going on there.  If you (or anyone else)
> believe that the work going on in OASIS is misguided you should comment
> there [1], not here.

While involved at OASIS I did comment. However, I am not now seeking to 
reform the AMQP work at OASIS. Any such effort should indeed take place 
there and not here.

Neither am I trying to subvert it. I have no issue with you or anyone 
else working on projects within Qpid related to specifications you are 
progressing at OASIS, providing anyone else can join in and contribute 
ideas.

All I am saying is that I don't consider OASIS to have authority over 
what other work goes on in Qpid, again provided that work is done in an 
open and collaborative manner which listens to criticism of the approach.

Collaboration doesn't have to mean that everyone agrees on one point of 
view. A degree of exploration of alternatives is in my view healthy at 
this point provided we all conform to the underlying specification as 
published and strive to ensure the various voices in the community are 
heard and that whatever is produced serves the needs of those using it.

I myself will very happily contribute to, align with and support any 
effort that I see emerging with genuine widespread support from existing 
communities. If that is a spec from OASIS, great. If it's an idea from 
one of the other implementers, great. I want any additional standard to 
be adopted on merit not by fiat.

In the end I believe we want the same thing, we have the same goal of 
richer interoperability and I do firmly believe that while in theory we 
have different views on how best to achieve that we can collaborate 
effectively on the practical side of aiding AMQP adoption and on 
concrete issues we are as often as not likely to find a happy compromise.

Now, as to improving the process at Qpid and building a more diverse set 
of contributors, that I am very much interested in and this is the 
perfect place to discuss it. I would be eager to hear from anyone with 
ideas that could help.

> While I may not be entirely happy with the OASIS process, the value in AMQP
> is in standardisation.

The value is in practical interoperability and interchangeability.

Though I personally don't think OASIS is the best route to achieving 
that, I respect your view that it is. There should be room for both of 
us to help users reap the benefits that AMQP promised.

>  If Qpid can be a place where multiple parties can
> come together and work together then it may be a good place for de-facto
> standards to emerge.

I want to be really clear on this point as it comes up several times 
here and in previous mails.

I said that I believe the emergence of de-facto standards is something 
to be nurtured and encouraged. I think Qpid can have a part in that, but 
I think it can *only* be a part. It *must* involve other communities, 
projects and products. So I am certainly *not* suggesting that I believe 
that work at Qpid should be taken as a de-facto standard.

[...]
> Historically Qpid has not been seen as a neutral place.  For better or
> worse, there are a preponderance of committers from one organisation.  In
> order to counter perceptions of a lack of neutrality I think work has to be
> done to bring more people in before we try to sell ourselves as a neutral
> venue for emerging standards.

Again, just so there is no misunderstanding, I am *not* 'selling Qpid' 
as such a venue...

>  Until we can actually demonstrate that Qpid
> is such a place I think it is presumptuous of us to try to claim de-facto
> standards for our work.

...and I have made no such claim.

A de-facto standard is not something you can claim. It is something you 
prove by showing the different interoperable implementations.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Rob Godfrey <ro...@gmail.com>]

On 10 October 2013 12:46, Gordon Sim <gs...@redhat.com> wrote:

> On 10/10/2013 10:58 AM, Rob Godfrey wrote:
>
>> I think the point of Qpid (vs. any other messaging
>> implementation at Apache or elsewhere) is to implement the AMQP
>> specification.
>>
>
> I have no disagreement when the AMQP specification is what is currently
> published.
>
> My concern is where that is defined to be an expanding, all-encompassing
> effort that ring fences whole areas of behaviour and sets itself up as the
> only legitimate avenue for exploration.
>

Do you believe that is what the OASIS TCs are doing?  If so in which areas?


>

Expanding beyond the current specification needs a more diverse source of
> ideas and a more open, transparent and collaborative process.
>
>
While I agree that a more open process is most definitely desirable, I
would argue that the OASIS TCs (sadly) currently have a much more diverse
membership than Qpid committers.  Also note that you do not need to be a
member of OASIS to comment on work going on there.  If you (or anyone else)
believe that the work going on in OASIS is misguided you should comment
there [1], not here.

While I may not be entirely happy with the OASIS process, the value in AMQP
is in standardisation.  If Qpid can be a place where multiple parties can
come together and work together then it may be a good place for de-facto
standards to emerge.


> [...]
>
>  I think that any such efforts at de-facto standardisation
>> must first reach out to other AMQP implementers and ensure there is a
>> broad
>> agreement on direction.  If the Qpid project can be a vehicle for doing
>> this, then great - however currently that is not how Qpid is operating and
>> I would be very concerned at us trying to claim any sort of work done
>> within Qpid as a "de-facto" standard.
>>
>
> Indeed and that of course is a straw man since no one is suggesting that
> we claim any such thing.
>
> However I would be very concerned if the OASIS TCs, with such limited
> representation, were enshrined as authorities over any and every aspect of
> work they choose to start.
>
> At present there is sadly no AMQP-wide community. While Qpid is far from
> perfect, it is at least, as an Apache project, founded on the  ethos of
> open, community driven collaboration. It has rules for governance that
> provide the means to correct deviations from that. Everything we do should
> be open and subject to debate and consensus.
>

Indeed... I would very much like to have an AMQP-wide community
established.  If Qpid can become the home of that then that is great,
however in order to facilitate that then I believe the first step would be
to be good AMQP citizens and work with and implement the emerging standards
that are currently being progressed (or be clear in our objections to said
work).

Historically Qpid has not been seen as a neutral place.  For better or
worse, there are a preponderance of committers from one organisation.  In
order to counter perceptions of a lack of neutrality I think work has to be
done to bring more people in before we try to sell ourselves as a neutral
venue for emerging standards.  Until we can actually demonstrate that Qpid
is such a place I think it is presumptuous of us to try to claim de-facto
standards for our work.  With greatest respect to those involved, that is
the way that leads to the next QMF.

I want users of Qpid to be assured that components or libraries that they
can get here will work interoperably with any AMQP implementation that
follows the standards.


>
> Reaching out to - and collaborating with - other implementations and
> communities is something I personally feel we must do more of in order to
> realise the promise behind AMQP. Starting those discussions here seems like
> one (though certainly not the only) reasonable way to begin however.
>
>
>
-- Rob

[1] https://www.oasis-open.org/committees/comments/index.php?wg_abbrev=amqp


> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: Qpid Dispatch Router component

[Gordon Sim <gs...@redhat.com>]

On 10/10/2013 10:58 AM, Rob Godfrey wrote:
> I think the point of Qpid (vs. any other messaging
> implementation at Apache or elsewhere) is to implement the AMQP
> specification.

I have no disagreement when the AMQP specification is what is currently 
published.

My concern is where that is defined to be an expanding, all-encompassing 
effort that ring fences whole areas of behaviour and sets itself up as 
the only legitimate avenue for exploration.

Expanding beyond the current specification needs a more diverse source 
of ideas and a more open, transparent and collaborative process.

[...]
> I think that any such efforts at de-facto standardisation
> must first reach out to other AMQP implementers and ensure there is a broad
> agreement on direction.  If the Qpid project can be a vehicle for doing
> this, then great - however currently that is not how Qpid is operating and
> I would be very concerned at us trying to claim any sort of work done
> within Qpid as a "de-facto" standard.

Indeed and that of course is a straw man since no one is suggesting that 
we claim any such thing.

However I would be very concerned if the OASIS TCs, with such limited 
representation, were enshrined as authorities over any and every aspect 
of work they choose to start.

At present there is sadly no AMQP-wide community. While Qpid is far from 
perfect, it is at least, as an Apache project, founded on the  ethos of 
open, community driven collaboration. It has rules for governance that 
provide the means to correct deviations from that. Everything we do 
should be open and subject to debate and consensus.

Reaching out to - and collaborating with - other implementations and 
communities is something I personally feel we must do more of in order 
to realise the promise behind AMQP. Starting those discussions here 
seems like one (though certainly not the only) reasonable way to begin 
however.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Rob Godfrey <ro...@gmail.com>]

On 10 October 2013 11:49, Gordon Sim <gs...@redhat.com> wrote:

> On 10/10/2013 09:38 AM, Rob Godfrey wrote:
>
>> My main concern is that I believe that Qpid should be primarily directed
>> at
>> implementing AMQP standards, and building resuable toolkits and components
>> that fit into any AMQP network.  I'd be very concerned if we were
>> inventing
>> alternative management protocols, or building components that only
>> interoperated with other Qpid tools.
>>
>> So, personally I'd like to see statement around components saying that
>> they
>> will be fully implementing emerging AMQP Management, AMQP addressing, etc.
>> standards, and that we as a project then ensure we stick to these goals.
>>
>
> Interoperability with other AMQP compliant components both in and out of
> Qpid and Apache is certainly key. That is what AMQP is all about and what
> Qpid should be about.
>
> Faithful implementation of the existing AMQP specification is clearly a
> requirement as that is central to the charter of the project. Where any
> auxiliary or emerging specifications are adopted by a component, whether
> they are AMQP related or not, they should be compliant with that.
>
> However, having a general policy where all Qpid components are required to
> implement whatever 'emerging standards' come out of the OASIS AMQP TCs is
> not something I am comfortable with.
>

Saying that all Qpid components implement all emerging standards is clearly
not what I am saying, because not all standards are appropriate for all
components.  However I think the point of Qpid (vs. any other messaging
implementation at Apache or elsewhere) is to implement the AMQP
specification.  I'd generally question why work was being carried out in
Qpid (as opposed to elsewhere) if the work is not based on existing or
emerging AMQP standards.


>
> The Qpid community as a whole needs to have a say in how future features
> will work through an open, collaborative process (open to _anyone_, even
> those primarily involved with other AMQP related projects or products).
>
>
i completely agree.


> Personally I think this would be better for AMQP as well. Allowing and
> encouraging the emergence of grass-roots driven, de-facto 'standards'
> developed in and between open, collaborative and transparent communities
> and backed up by proven interoperable implementations, which can then form
> the basis for official de-jure standardisation.
>
>
Possibly, but I think that any such efforts at de-facto standardisation
must first reach out to other AMQP implementers and ensure there is a broad
agreement on direction.  If the Qpid project can be a vehicle for doing
this, then great - however currently that is not how Qpid is operating and
I would be very concerned at us trying to claim any sort of work done
within Qpid as a "de-facto" standard.  Where there is qork going on in AMQP
then we as the Qpid community should be ensuring that we feed back to that
and raise questions/objections as necessary (whether we are part of the
OASIS group or not - feedback from the public is possible).

-- Rob

------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: Qpid Dispatch Router component

[Gordon Sim <gs...@redhat.com>]

On 10/10/2013 09:38 AM, Rob Godfrey wrote:
> My main concern is that I believe that Qpid should be primarily directed at
> implementing AMQP standards, and building resuable toolkits and components
> that fit into any AMQP network.  I'd be very concerned if we were inventing
> alternative management protocols, or building components that only
> interoperated with other Qpid tools.
>
> So, personally I'd like to see statement around components saying that they
> will be fully implementing emerging AMQP Management, AMQP addressing, etc.
> standards, and that we as a project then ensure we stick to these goals.

Interoperability with other AMQP compliant components both in and out of 
Qpid and Apache is certainly key. That is what AMQP is all about and 
what Qpid should be about.

Faithful implementation of the existing AMQP specification is clearly a 
requirement as that is central to the charter of the project. Where any 
auxiliary or emerging specifications are adopted by a component, whether 
they are AMQP related or not, they should be compliant with that.

However, having a general policy where all Qpid components are required 
to implement whatever 'emerging standards' come out of the OASIS AMQP 
TCs is not something I am comfortable with.

The Qpid community as a whole needs to have a say in how future features 
will work through an open, collaborative process (open to _anyone_, even 
those primarily involved with other AMQP related projects or products).

Personally I think this would be better for AMQP as well. Allowing and 
encouraging the emergence of grass-roots driven, de-facto 'standards' 
developed in and between open, collaborative and transparent communities 
and backed up by proven interoperable implementations, which can then 
form the basis for official de-jure standardisation.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Rob Godfrey <ro...@gmail.com>]

On 10 October 2013 20:07, Ted Ross <tr...@redhat.com> wrote:

>
> On 10/10/2013 11:20 AM, Rob Godfrey wrote:
>
>> On 10 October 2013 15:35, Ted Ross <tr...@redhat.com> wrote:
>>
>>  On 10/10/2013 04:38 AM, Rob Godfrey wrote:
>>>
>>>
>
[...]


> So here's the root of the issue.  The question is whether or not Dispatch
> is too tangential in its relation to AMQP to be considered appropriate to
> be hosted in Apache Qpid.
>
>
To me that's *really* not the root of the issue at all.  The issue to me is
that I don't know what Dispatch is, how it differs in scope and vision from
a broker or other component, and how it fits into the Qpid story.  In
practice I probably do know because of off-list conversations, but I
couldn't get that information from the mails or documentation here.  I
think if we were to include the definition/description that Gordon gave
then we would have a really good scope statement on what it was about.


> You stated later in your email that you don't know what an AMQP-compliant
> router is.  Well, nobody really does because nothing like it has ever
> existed.  The very idea was not even conceivable before there was an AMQP
> 1.0.  But now we have AMQP and a door has been opened to create really
> interesting solutions to difficult problems in large-scale distributed
> computing. Dispatch is an early attempt to implement such a solution.  If
> we decide that by "tangential" we mean "beyond the scope of traditional
> broker-based messaging", then Dispatch is clearly tangential and I will go
> and find another community to host it.
>

Again this is really not what I'm saying.  As Fraser's mail indicates there
is currently no clarity in how the various Qpid components fit together.
What I want is for us to improve that by having a clear and *shared*
understanding of what all the components are.


>
> The only way to know if a project will have long-term value is to let it
> run for awhile.  It's a form of incubation.  We need a way, as you suggest,
> of deciding what to incubate.  We then need a way to decide when a
> sub-project has failed and should be abandoned. Given that we have no such
> guidelines in place, I intend to go forward with the same process we used
> for the JMS project.  If there arises a consensus in the community that
> Dispatch does not, or might not belong in Qpid, I will not go forward.
>
> In the mean time, I will try to fill in some of the gaps in information
> that you've identified.
>
>
So, to reiterate I am in no way suggesting that Dispatch shouldn't be a
component in Qpid... I actually think it sounds like a really good idea and
something we should be doing.  However I think we should be aiming at
better explaining to ourselves and others what the goals behind our
components are.

i do think we were remiss in not setting up a proper scope statement for
the JMS client.  I'd actually suggest that we come up with one now and put
it to a vote to ensure that we have broad agreement on the proposed
component and to discover now if there are any misconceptions/disagreements
about the form that work should take.  I'll see if Robbie and I can come up
with something on that one (though obviously others are welcome to do the
work there instead :-) ).

Cheers,
Rob


> -Ted
>
>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: Qpid Dispatch Router component

[Ted Ross <tr...@redhat.com>]

On 10/10/2013 11:20 AM, Rob Godfrey wrote:
> On 10 October 2013 15:35, Ted Ross <tr...@redhat.com> wrote:
>
>> On 10/10/2013 04:38 AM, Rob Godfrey wrote:
>>
>>> My main concern is that I believe that Qpid should be primarily directed
>>> at implementing AMQP standards, and building resuable toolkits and
>>> components that fit into any AMQP network. I'd be very concerned if we were
>>> inventing alternative management protocols, or building components that
>>> only interoperated with other Qpid tools. So, personally I'd like to see
>>> statement around components saying that they will be fully implementing
>>> emerging AMQP Management, AMQP addressing, etc. standards, and that we as a
>>> project then ensure we stick to these goals.
>>>
>> Rob,
>>
>> Here's where I'm going to have to disagree with you in principle. I
>> believe that Qpid should be primarily directed at innovating with AMQP and
>> helping to drive the AMQP standards where appropriate.  If Qpid doesn't do
>> it, somebody else will.  I should point out that almost everything we do
>> here is well ahead of the standards, including the JMS client.
>>
>> The thing I object to in your statement is the direction of flow from the
>> standard to the implementation.  Standards bodies _do not innovate_.  If
>> the emerging standards are such that a particular valuable innovation
>> cannot be done, the standard needs to change or be ignored.  Qpid must not
>> allow itself to be put in the position of meekly sitting and waiting for
>> the tablets to come from on high before implementing.
>>
>>
> I'm not saying that there is a direction of standard -> implementation, but
> that if there is a standard we should be implementing it rather than
> rolling our own which conflicts or substantially overlaps.  If we see a
> standard emerging at OASIS and do not believe it is correct then we should
> be working to fix it at that venue.  If we do work which we think is
> generally applicable to other AMQP implementations then we should be
> considering whether we want to push it as a standard at OASiS or elsewhere.
>
> I absolutely don't think that Qpid should be restricted in scope to simply
> implementing the standards, and I strongly believe that Qpid can be a place
> where we innovate and then work towards bringing behaviours to a standards
> body.  However I also think that when we do so we should state up front
> what it is we are trying to achieve.  I also don't think that qpid should
> become a mini-GitHub for any project that is tangentially related to AMQP
> to simply use as a code repository.

So here's the root of the issue.  The question is whether or not 
Dispatch is too tangential in its relation to AMQP to be considered 
appropriate to be hosted in Apache Qpid.

You stated later in your email that you don't know what an 
AMQP-compliant router is.  Well, nobody really does because nothing like 
it has ever existed.  The very idea was not even conceivable before 
there was an AMQP 1.0.  But now we have AMQP and a door has been opened 
to create really interesting solutions to difficult problems in 
large-scale distributed computing. Dispatch is an early attempt to 
implement such a solution.  If we decide that by "tangential" we mean 
"beyond the scope of traditional broker-based messaging", then Dispatch 
is clearly tangential and I will go and find another community to host it.

The only way to know if a project will have long-term value is to let it 
run for awhile.  It's a form of incubation.  We need a way, as you 
suggest, of deciding what to incubate.  We then need a way to decide 
when a sub-project has failed and should be abandoned. Given that we 
have no such guidelines in place, I intend to go forward with the same 
process we used for the JMS project.  If there arises a consensus in the 
community that Dispatch does not, or might not belong in Qpid, I will 
not go forward.

In the mean time, I will try to fill in some of the gaps in information
that you've identified.

-Ted



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Rob Godfrey <ro...@gmail.com>]

So, just to be clear the reason I asked the question is to try to help us
frame better a scope statement.

People often ask why we currently have both a Java Broker and a C++ broker,
and I think it's a fair question :-).  If we start building up Dispatch as
a component and someone says "hey - it'd be really cool to have some sort
of message store built into Dispatch" and so on... how do we define the
difference between what Dispatch might become and what we have in brokers
currently?  The questions on client APIs were really a nod to the confusion
in scope that I see in Proton.  Again, within Messenger I see it as not a
huge step to suddenly add some storage and a bit of configuration and again
... you have something that looks a lot like a broker.  Going back to
Fraser's mail I think we need to be clear on what each component is and
what it isn't (being clear on it ourselves seems to be a pre-requisite of
being able to explain it to others :-) ).

-- Rob

On 11 October 2013 11:54, Gordon Sim <gs...@redhat.com> wrote:

> I don't claim to speak with any authority on Dispatch Router, but fwiw,
> here's my answer to these two specific questions:
>
>
> On 10/10/2013 04:20 PM, Rob Godfrey wrote:
>
>> How does a router differ from a broker?
>>
>
> I tend to have a more general view of what 'broker' might mean, but in the
> context of statements that Dispatch Router 'is not a broker', what is meant
> is the type of intermediary as exemplified by the two existing Qpid brokers
> and others like them.
>
> As I see it, there are two key differences between Dispatch Router and
> these.
>
> The first is that a principle of the Dispatch Router is that it does not
> 'take ownership' of messages. This is in contrast to most brokers where the
> publisher transfers responsibility for messages to the broker and the
> broker then undertakes to deliver them allowing the producer to forget
> about them. With Dispatch Router, the idea is that it will route those
> messages, but any guarantee regarding acceptance of those messages comes
> from the consumer(s) of the message, not from the intermediary.
>
> The second distinction is that Dispatch Router is from the start designed
> to be a small piece in a distributed routing network of interconnected
> components.
>
> Now, there is nothing that says a broker can't do these things. In fact I
> added code a while back to the Qpid c++ broker that does the former -
> relays messages through it, providing end-to-end guarantees.
>
> Likewise the distributed architecture has its roots in things like the
> federation in the Qpid c++ broker (and similar solutions by other brokers).
>
> The Router however is setting out from the start to do *only* these
> things. It is based on the idea that these are separate concerns that can
> be split apart allowing 'brokers' to focus on queueing, persistence,
> transactions, augmented functionality such as LVQ patterns etc, and that
> the 'federation' logic can be provided in a separate component that can
> hook up different brokers into a federation as well as hooking in other
> types of AMQP based services and allow clients to connect directly to the
> router as well.
>
> By focusing on one aspect its thought components can do a better job. E.g.
> as I understand it, the Dispatch Router aims to address some of the
> weaknesses of the federation support in qpidd, by providing for redundant,
> fault-tolerant routes without duplication of messages, adapting to failures
> and dynamically computing the best path between two endpoints.
>
> I confess, for all its faults, federation has been one of my favourite
> features of the c++ broker because of the potential I see in it. That is
> why I started off adding some 1.0 related improvements. However I think
> Ted's thinking on Dispatch Router is quite compelling (though I also feel a
> lot of the necessary detail is still missing and it still needs to be
> proven).
>
> For my part, I am excited to see how this develops, am keen to take part
> if I can and would encourage anyone with an interest to chip in with ideas,
> feedback, criticism, questions, code, use cases or whatever else. This
> began as Ted's 'brainchild' - and most things begin as someone's idea - but
> it really is now open to anyone and everyone who is interested. Get
> involved and shape the direction!
>
>
>  Would you expect any special client APIs to form part of the router
>> package or not?
>>
>
> I'm not entirely sure if I understand the question, but there are again
> two points I would make.
>
> First, in my opinion it is *vital* that no special client is required to
> use Dispatch Router effectively. Transparency is a key property. You should
> be able to point a JMS application or qpid::messaging or any other
> compliant AMQP client at it. Any special coupling between this and proton
> based clients would in my view be a horrendous mistake. (Nothing like that
> is planned I'm sure, I'm just using this as a hypothetical example to make
> the point).
>
> Second, the code in Dispatch Router is in theory designed around a toolkit
> for building  AMQP 'containers' of different kinds, with the router being
> one such example (another might be a proxy focused more on enforcing ACLs
> at the edge). In theory this could be viewed as an API of sorts. However I
> think at this point its better viewed as a sensible desire for some
> internal structure and separation of concerns. A publishable 'API' is in my
> view some way off and would require a lot of work that would at this point
> distract from the main goal, which is to define the behaviour of the router
> and implement it.
>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: Qpid Dispatch Router component

[Gordon Sim <gs...@redhat.com>]

On 10/12/2013 11:40 AM, Fraser Adams wrote:
> a primary concern and motivation has to be standardisation,
> particularly Open Standards and they must be seen to be platform
> neutral - so fundamentally AMQP first and Qpid second. To me that's a
> reasonable position too because, as has been expressed elsewhere, one
> of the compelling reasons of "why AMQP" is because it's an Open
> Standard and *should be* interoperable.

I don't disagree with any of that. Compliance with the current 
specification and a commitment to demonstrable interoperability around 
that is, and must be, fundamental to Qpid.

I do also very much want to see interoperability extend to areas not 
directly covered by this specification.

What I am uncomfortable with is the view that any initiative, wherever 
and however it was begun, has from the start some unassailable dominion 
over the area it stakes out for itself.

> Sadly that wasn't really achieved for 0.10, but it certainly seems to
> be the case for 1.0 which I feel we ought to embrace with a passion.

The history of the 0-10 specification is actually interesting to 
consider here. Though it was the de-jure standard, having passed by 
unanimous vote through the old working group process, it was clear from 
the start that there was no real consensus behind it (even from some of 
those voting, not to mention anyone not represented in the process). 
Though supported by all Qpid components, it was implemented by nothing 
else. That is a pattern I don't want to see repeated.

While achieving real consensus is hard, slow and often frustrating, 
'official' standards that aren't widely supported are of limited value.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Fraser Adams <fr...@blueyonder.co.uk>]

I'd like to wade in slightly here if I may as I've been following this 
thread with great interest.

I guess that I might be in a vaguely interesting position as although my 
employers are interested in and make use of AMQP/Qpid my contributions 
to this group have been entirely at a personal level and not on behalf 
of an employer, simply because I think it's interesting and worthwhile.

The reason I mention that is in relation to organisations like OASIS.

So I find myself rather torn, I really agree with Gordon and Ted on one 
hand about innovation and driving things bottom up (decent standards 
tend to come with "reference implementations") but on the other hand 
I've got a lot of sympathy with Rob and his colleagues who very much 
have a position that a primary concern and motivation has to be 
standardisation, particularly Open Standards and they must be seen to be 
platform neutral - so fundamentally AMQP first and Qpid second. To me 
that's a reasonable position too because, as has been expressed 
elsewhere, one of the compelling reasons of "why AMQP" is because it's 
an Open Standard and *should be* interoperable. Sadly that wasn't really 
achieved for 0.10, but it certainly seems to be the case for 1.0 which I 
feel we ought to embrace with a passion.

But why I mention OASIS, well TBH I have a bit of a problem with them in 
the sense that they are putting together Open Standards for sure, but I 
don't really believe that the process for achieving said standards seems 
to be in any way open and transparent!!

One of the reasons that I like Apache is the fundamental doctrine of 
openness and having a licence that is non-restrictive, so products can 
be used most anywhere. We're in a position where organisations and 
individuals can contribute and is essentially democratic/meritocratic - 
that feels like a healthy place to be in the 21st Century.

The same can't be said of OASIS, so someone like myself just might have 
vaguely relevant experiences with management say, but there's no real 
avenue for contribution because it's a bit of a "closed shop" made up of 
fairly large organisations who have to pay a non-trivial fee for 
membership (I'm keen, but I'm not *that* keen :-)). Now I can sympathise 
a bit and I'm sure that part of the motivation is paranoia about 
avoidance of potential submarine patents probably driven by some of the 
big players, but I can't help believe that there might be better and 
more open ways to achieve the same goals (those guys can afford some 
decent lawyers after all :-)).

One of the things about OASIS that worries me as an outsider is how much 
of the decision making ends up getting driven by some corporate agenda - 
after all if you've got a product to sell there might be benefits in 
shaping the direction of a standard to best suit your architecture or 
roadmap and potentially give some commercial advantage. I'm not accusing 
any organisation of doing such a thing and I'm sure that it's all lovely 
and altruistic, but without openness and transparency it's hard to say 
for sure. If I'm honest given where AMQP sits in the protocol stack I 
often wonder why the hell it's OASIS and not say IETF that holds the 
standard, from what I can see OASIS seems to sit rather more at the 
"higher end of the stack" formalising B2B interoperability whereas AMQP 
1.0 seems to have evolved into something arguably closer to a network 
layer protocol (after all we're on a thread about routers :-)).

Anyway this probably hasn't added much to the discussion and has turned 
a bit into a polemic about openness and transparency so sorry about 
that, but I'm quite passionate that it's healthier for an Open Source 
model to encompass more than just "a bit of code" and I think that when 
we're talking about Open Standards the Open Source model should extend 
out to ideas and architectures as well - you've probably noted from my 
"how it all hangs together" post that I'm keen for a visible, open, 
modular architecture :-)

Bottom line is what  I'd *really* like to see is Qpid components working 
towards and enabling Open Standards and I'd like to see OASIS opening up 
a bit and being more practical. FWIW I'm a little nervous about the 
Management Specification and probably won't feel happy until we've got 
something kickable, I've read it a couple of times and it still feels a 
little abstract and open to interpretation, but that's largely in the 
nature of these things, which is why reference implementations are good 
and should help the standards evolve.

Regards,
Frase


On 11/10/13 21:36, Gordon Sim wrote:
> On 10/11/2013 06:33 PM, William Henry wrote:
>> Yes Ted, and Gordon I believe, has been involved in trying to get 
>> this feedback based on our experiences.
>
> Simplify to clarify: I am no longer involved in any way with OASIS.
>
> Further, any opinions I expressed while I was there were my own and 
> did not in any way represent the views of my employer or any of my 
> colleagues.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Gordon Sim <gs...@redhat.com>]

On 10/11/2013 06:33 PM, William Henry wrote:
> Yes Ted, and Gordon I believe, has been involved in trying to get this feedback based on our experiences.

Simplify to clarify: I am no longer involved in any way with OASIS.

Further, any opinions I expressed while I was there were my own and did 
not in any way represent the views of my employer or any of my colleagues.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[William Henry <wh...@redhat.com>]

Thanks Steve,

Yes Ted, and Gordon I believe, has been involved in trying to get this feedback based on our experiences. 

As you know I was involved in the TC for a while but I got dragged elsewhere. Also, perhaps I'm not as patient as I used to be for standards body work. It's a tough job and requires a lot of patience. Maybe too much ;-)

William

----- Original Message -----
> Hi William,
> 
> Thanks for expressing your view. And no reasonable person would discount it.
> 
> Speaking for my own points, I would not argue that Dispatch (or any
> trailblazing development) should _wait_ for OASIS. However, since there are
> 2 areas of active work in OASIS that are related, and Dispatch is quickly
> gaining experience and backing, I would very much like to see Dispatch
> development folks actively involved in informing the OASIS technical
> committees of their experience and help to get the eventual standard(s)
> worked out in a way that's been proven in the field.
> 
> -Steve
> 
> > -----Original Message-----
> > From: William Henry [mailto:whenry@redhat.com]
> > Sent: Friday, October 11, 2013 12:49 PM
> > To: users@qpid.apache.org
> > Subject: Re: Qpid Dispatch Router component
> > 
> > 
> > I'd also add, because I'm sure that some of you wonder why is William
> > weighing in? What has he to do with AMQP/Qpid anyway besides being
> > historically linked to it?
> > 
> > I've travelled globally talking about AMQP and Qpid to investment banks,
> > government agencies, telecoms, animation studios, stock exchanges etc. etc.
> > (strangers on planes ;-)  I'm sure others have done more to help promote
> > AMQP and Qpid but, despite my seemingly lack of, or invisible, community
> > involvement of late, I bet that list of others-that-have-done-more-to-
> > promote it is actually not a very large list. I can think of a few. I have
> > other
> > technology areas I focus on but I continue to be involved in AMQP/Qpid
> > because I get pulled back in, and I still "believe". There is a massive
> > opportunity for AMQP that is more than just simple MOM.
> > 
> > So my 2 cents at a high level:
> > 1) People love the AMQP story
> > 2) People love the AMQP network story - with a variety of AMQP based
> > assets.
> > 3) As we've introduced the concept of what Dispatch does, people love it!
> > Did I say love it?! And they do not see it as tangential to AMQP. But
> > massively
> > complimentary. A Dispatch type network with (various) brokers backing it up
> > nearer to applications - that have different client interfaces (JMS, JCA,
> > Python, C/C++, Ruby etc.)
> > 4) The cloud will and is already playing a big part in AMQP adoption. We
> > need
> > to capitalize.
> > 5) People want solutions now! Management now. Addressing nailed down
> > now.
> > 
> > Best,
> > William
> > 
> > ----- Original Message -----
> > > +1. Great post Gordon. Again something to be captured in Qpid
> > > +community
> > > pages/documentation.
> > >
> > > I'd also add we need to be innovating fast around AMQP now that we have
> > 1.0.
> > > We can't always wait on the OASIS process. It has taken a very long
> > > time to get here (not OASIS). There were some significant changes along
> > the way.
> > > Users are looking for solutions today.
> > >
> > > I understand there is a risk of getting ahead of ourselves in certain
> > > areas and having to backtrack. However people that have been investing
> > > in AMQP are looking for solutions to their problems ASAP and can't
> > > wait until every area has been worked out.
> > >
> > > I agree there is a risk of a high cost to "fixing" addressing and
> > > management.
> > > How can we get there faster? Surely innovations/projects like Dispatch
> > > can only help in driving the standardization faster as it and other
> > > artifacts hit issues.  I think Dispatch has already helped drive the
> > > addressing discussion.
> > >
> > > Let's press forward.
> > >
> > > William
> > >
> > > ----- Original Message -----
> > > > I don't claim to speak with any authority on Dispatch Router, but
> > > > fwiw, here's my answer to these two specific questions:
> > > >
> > > > On 10/10/2013 04:20 PM, Rob Godfrey wrote:
> > > > > How does a router differ from a broker?
> > > >
> > > > I tend to have a more general view of what 'broker' might mean, but
> > > > in the context of statements that Dispatch Router 'is not a broker',
> > > > what is meant is the type of intermediary as exemplified by the two
> > > > existing Qpid brokers and others like them.
> > > >
> > > > As I see it, there are two key differences between Dispatch Router
> > > > and these.
> > > >
> > > > The first is that a principle of the Dispatch Router is that it does
> > > > not 'take ownership' of messages. This is in contrast to most
> > > > brokers where the publisher transfers responsibility for messages to
> > > > the broker and the broker then undertakes to deliver them allowing
> > > > the producer to forget about them. With Dispatch Router, the idea is
> > > > that it will route those messages, but any guarantee regarding
> > > > acceptance of those messages comes from the consumer(s) of the
> > message, not from the intermediary.
> > > >
> > > > The second distinction is that Dispatch Router is from the start
> > > > designed to be a small piece in a distributed routing network of
> > > > interconnected components.
> > > >
> > > > Now, there is nothing that says a broker can't do these things. In
> > > > fact I added code a while back to the Qpid c++ broker that does the
> > > > former - relays messages through it, providing end-to-end guarantees.
> > > >
> > > > Likewise the distributed architecture has its roots in things like
> > > > the federation in the Qpid c++ broker (and similar solutions by other
> > brokers).
> > > >
> > > > The Router however is setting out from the start to do *only* these
> > > > things. It is based on the idea that these are separate concerns
> > > > that can be split apart allowing 'brokers' to focus on queueing,
> > > > persistence, transactions, augmented functionality such as LVQ
> > > > patterns etc, and that the 'federation' logic can be provided in a
> > > > separate component that can hook up different brokers into a
> > > > federation as well as hooking in other types of AMQP based services
> > > > and allow clients to connect directly to the router as well.
> > > >
> > > > By focusing on one aspect its thought components can do a better job.
> > > > E.g. as I understand it, the Dispatch Router aims to address some of
> > > > the weaknesses of the federation support in qpidd, by providing for
> > > > redundant, fault-tolerant routes without duplication of messages,
> > > > adapting to failures and dynamically computing the best path between
> > > > two endpoints.
> > > >
> > > > I confess, for all its faults, federation has been one of my
> > > > favourite features of the c++ broker because of the potential I see
> > > > in it. That is why I started off adding some 1.0 related
> > > > improvements. However I think Ted's thinking on Dispatch Router is
> > > > quite compelling (though I also feel a lot of the necessary detail
> > > > is still missing and it still needs to be proven).
> > > >
> > > > For my part, I am excited to see how this develops, am keen to take
> > > > part if I can and would encourage anyone with an interest to chip in
> > > > with ideas, feedback, criticism, questions, code, use cases or whatever
> > else.
> > > > This began as Ted's 'brainchild' - and most things begin as
> > > > someone's idea - but it really is now open to anyone and everyone
> > > > who is interested. Get involved and shape the direction!
> > > >
> > > > > Would you expect any special client APIs to form part of the
> > > > > router package or not?
> > > >
> > > > I'm not entirely sure if I understand the question, but there are
> > > > again two points I would make.
> > > >
> > > > First, in my opinion it is *vital* that no special client is
> > > > required to use Dispatch Router effectively. Transparency is a key
> > > > property. You should be able to point a JMS application or
> > > > qpid::messaging or any other compliant AMQP client at it. Any
> > > > special coupling between this and proton based clients would in my
> > > > view be a horrendous mistake. (Nothing like that is planned I'm
> > > > sure, I'm just using this as a hypothetical example to make the point).
> > > >
> > > > Second, the code in Dispatch Router is in theory designed around a
> > > > toolkit for building  AMQP 'containers' of different kinds, with the
> > > > router being one such example (another might be a proxy focused more
> > > > on enforcing ACLs at the edge). In theory this could be viewed as an
> > > > API of sorts. However I think at this point its better viewed as a
> > > > sensible desire for some internal structure and separation of
> > > > concerns. A publishable 'API' is in my view some way off and would
> > > > require a lot of work that would at this point distract from the
> > > > main goal, which is to define the behaviour of the router and implement
> > it.
> > > >
> > > > --------------------------------------------------------------------
> > > > - To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
> > > > additional commands, e-mail: users-help@qpid.apache.org
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
> > > additional commands, e-mail: users-help@qpid.apache.org
> > >
> > >
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For additional
> > commands, e-mail: users-help@qpid.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Rob Godfrey <ro...@gmail.com>]

On 11 October 2013 19:43, Ted Ross <tr...@redhat.com> wrote:

> Steve,
>
> I'm a member of the OASIS technical committee that is working on both the
> addressing and management specifications.  I see my role there as
> tester/implementer.  Both specifications have a long way to go before they
> are usable IMO.  It is my plan that Dispatch will become compliant with
> both specifications, but it is a two-way street.  The TC needs also to
> respect the needs of implementers.
>
>
I absolutely agree... that is the point of the TCs.  It would be good if
all of us who have questions, comments, etc raise them to the TC, and those
of us who are members ensure that these concerns/comments/questions are
acted upon and also fedback to this list.

None of the work in progress is solidified yet precisely because it needs
feedback from implementers and users.  The work won't be finalised until we
have interoperable implementations from different sources.

If you've raised concerns around the two pieces of work in question and you
do not think they have been adequately addressed, you should definitely
send a chaser e-mail.  If there is stuff you haven't raised yet, then
please do so.

To be clear, the latest working draft of the addressing specification is
here:

https://www.oasis-open.org/committees/document.php?document_id=50701&wg_abbrev=amqp

And the latest draft of the management specification is here:

https://www.oasis-open.org/committees/document.php?document_id=50101&wg_abbrev=amqp

Those who are not members of the TCs can raise comments/questions through
the public comment list (see:
https://www.oasis-open.org/committees/comments/index.php?wg_abbrev=amqp)

Hope this helps,
Rob

-Ted
>
>
>
> On 10/11/2013 01:18 PM, Steve Huston wrote:
>
>> Hi William,
>>
>> Thanks for expressing your view. And no reasonable person would discount
>> it.
>>
>> Speaking for my own points, I would not argue that Dispatch (or any
>> trailblazing development) should _wait_ for OASIS. However, since there are
>> 2 areas of active work in OASIS that are related, and Dispatch is quickly
>> gaining experience and backing, I would very much like to see Dispatch
>> development folks actively involved in informing the OASIS technical
>> committees of their experience and help to get the eventual standard(s)
>> worked out in a way that's been proven in the field.
>>
>> -Steve
>>
>>  -----Original Message-----
>>> From: William Henry [mailto:whenry@redhat.com]
>>> Sent: Friday, October 11, 2013 12:49 PM
>>> To: users@qpid.apache.org
>>> Subject: Re: Qpid Dispatch Router component
>>>
>>>
>>> I'd also add, because I'm sure that some of you wonder why is William
>>> weighing in? What has he to do with AMQP/Qpid anyway besides being
>>> historically linked to it?
>>>
>>> I've travelled globally talking about AMQP and Qpid to investment banks,
>>> government agencies, telecoms, animation studios, stock exchanges etc.
>>> etc.
>>> (strangers on planes ;-)  I'm sure others have done more to help promote
>>> AMQP and Qpid but, despite my seemingly lack of, or invisible, community
>>> involvement of late, I bet that list of others-that-have-done-more-to-
>>> promote it is actually not a very large list. I can think of a few. I
>>> have other
>>> technology areas I focus on but I continue to be involved in AMQP/Qpid
>>> because I get pulled back in, and I still "believe". There is a massive
>>> opportunity for AMQP that is more than just simple MOM.
>>>
>>> So my 2 cents at a high level:
>>> 1) People love the AMQP story
>>> 2) People love the AMQP network story - with a variety of AMQP based
>>> assets.
>>> 3) As we've introduced the concept of what Dispatch does, people love it!
>>> Did I say love it?! And they do not see it as tangential to AMQP. But
>>> massively
>>> complimentary. A Dispatch type network with (various) brokers backing it
>>> up
>>> nearer to applications - that have different client interfaces (JMS, JCA,
>>> Python, C/C++, Ruby etc.)
>>> 4) The cloud will and is already playing a big part in AMQP adoption. We
>>> need
>>> to capitalize.
>>> 5) People want solutions now! Management now. Addressing nailed down
>>> now.
>>>
>>> Best,
>>> William
>>>
>>> ----- Original Message -----
>>>
>>>> +1. Great post Gordon. Again something to be captured in Qpid
>>>> +community
>>>> pages/documentation.
>>>>
>>>> I'd also add we need to be innovating fast around AMQP now that we have
>>>>
>>> 1.0.
>>>
>>>> We can't always wait on the OASIS process. It has taken a very long
>>>> time to get here (not OASIS). There were some significant changes along
>>>>
>>> the way.
>>>
>>>> Users are looking for solutions today.
>>>>
>>>> I understand there is a risk of getting ahead of ourselves in certain
>>>> areas and having to backtrack. However people that have been investing
>>>> in AMQP are looking for solutions to their problems ASAP and can't
>>>> wait until every area has been worked out.
>>>>
>>>> I agree there is a risk of a high cost to "fixing" addressing and
>>>> management.
>>>> How can we get there faster? Surely innovations/projects like Dispatch
>>>> can only help in driving the standardization faster as it and other
>>>> artifacts hit issues.  I think Dispatch has already helped drive the
>>>> addressing discussion.
>>>>
>>>> Let's press forward.
>>>>
>>>> William
>>>>
>>>> ----- Original Message -----
>>>>
>>>>> I don't claim to speak with any authority on Dispatch Router, but
>>>>> fwiw, here's my answer to these two specific questions:
>>>>>
>>>>> On 10/10/2013 04:20 PM, Rob Godfrey wrote:
>>>>>
>>>>>> How does a router differ from a broker?
>>>>>>
>>>>> I tend to have a more general view of what 'broker' might mean, but
>>>>> in the context of statements that Dispatch Router 'is not a broker',
>>>>> what is meant is the type of intermediary as exemplified by the two
>>>>> existing Qpid brokers and others like them.
>>>>>
>>>>> As I see it, there are two key differences between Dispatch Router
>>>>> and these.
>>>>>
>>>>> The first is that a principle of the Dispatch Router is that it does
>>>>> not 'take ownership' of messages. This is in contrast to most
>>>>> brokers where the publisher transfers responsibility for messages to
>>>>> the broker and the broker then undertakes to deliver them allowing
>>>>> the producer to forget about them. With Dispatch Router, the idea is
>>>>> that it will route those messages, but any guarantee regarding
>>>>> acceptance of those messages comes from the consumer(s) of the
>>>>>
>>>> message, not from the intermediary.
>>>
>>>> The second distinction is that Dispatch Router is from the start
>>>>> designed to be a small piece in a distributed routing network of
>>>>> interconnected components.
>>>>>
>>>>> Now, there is nothing that says a broker can't do these things. In
>>>>> fact I added code a while back to the Qpid c++ broker that does the
>>>>> former - relays messages through it, providing end-to-end guarantees.
>>>>>
>>>>> Likewise the distributed architecture has its roots in things like
>>>>> the federation in the Qpid c++ broker (and similar solutions by other
>>>>>
>>>> brokers).
>>>
>>>> The Router however is setting out from the start to do *only* these
>>>>> things. It is based on the idea that these are separate concerns
>>>>> that can be split apart allowing 'brokers' to focus on queueing,
>>>>> persistence, transactions, augmented functionality such as LVQ
>>>>> patterns etc, and that the 'federation' logic can be provided in a
>>>>> separate component that can hook up different brokers into a
>>>>> federation as well as hooking in other types of AMQP based services
>>>>> and allow clients to connect directly to the router as well.
>>>>>
>>>>> By focusing on one aspect its thought components can do a better job.
>>>>> E.g. as I understand it, the Dispatch Router aims to address some of
>>>>> the weaknesses of the federation support in qpidd, by providing for
>>>>> redundant, fault-tolerant routes without duplication of messages,
>>>>> adapting to failures and dynamically computing the best path between
>>>>> two endpoints.
>>>>>
>>>>> I confess, for all its faults, federation has been one of my
>>>>> favourite features of the c++ broker because of the potential I see
>>>>> in it. That is why I started off adding some 1.0 related
>>>>> improvements. However I think Ted's thinking on Dispatch Router is
>>>>> quite compelling (though I also feel a lot of the necessary detail
>>>>> is still missing and it still needs to be proven).
>>>>>
>>>>> For my part, I am excited to see how this develops, am keen to take
>>>>> part if I can and would encourage anyone with an interest to chip in
>>>>> with ideas, feedback, criticism, questions, code, use cases or whatever
>>>>>
>>>> else.
>>>
>>>> This began as Ted's 'brainchild' - and most things begin as
>>>>> someone's idea - but it really is now open to anyone and everyone
>>>>> who is interested. Get involved and shape the direction!
>>>>>
>>>>>  Would you expect any special client APIs to form part of the
>>>>>> router package or not?
>>>>>>
>>>>> I'm not entirely sure if I understand the question, but there are
>>>>> again two points I would make.
>>>>>
>>>>> First, in my opinion it is *vital* that no special client is
>>>>> required to use Dispatch Router effectively. Transparency is a key
>>>>> property. You should be able to point a JMS application or
>>>>> qpid::messaging or any other compliant AMQP client at it. Any
>>>>> special coupling between this and proton based clients would in my
>>>>> view be a horrendous mistake. (Nothing like that is planned I'm
>>>>> sure, I'm just using this as a hypothetical example to make the point).
>>>>>
>>>>> Second, the code in Dispatch Router is in theory designed around a
>>>>> toolkit for building  AMQP 'containers' of different kinds, with the
>>>>> router being one such example (another might be a proxy focused more
>>>>> on enforcing ACLs at the edge). In theory this could be viewed as an
>>>>> API of sorts. However I think at this point its better viewed as a
>>>>> sensible desire for some internal structure and separation of
>>>>> concerns. A publishable 'API' is in my view some way off and would
>>>>> require a lot of work that would at this point distract from the
>>>>> main goal, which is to define the behaviour of the router and implement
>>>>>
>>>> it.
>>>
>>>> ------------------------------**------------------------------**
>>>>> --------
>>>>> - To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>For
>>>>> additional commands, e-mail: users-help@qpid.apache.org
>>>>>
>>>>>
>>>>>  ------------------------------**------------------------------**
>>>> ---------
>>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>For
>>>> additional commands, e-mail: users-help@qpid.apache.org
>>>>
>>>>
>>>>  ------------------------------**------------------------------**
>>> ---------
>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>For additional
>>> commands, e-mail: users-help@qpid.apache.org
>>>
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: Qpid Dispatch Router component

[Ted Ross <tr...@redhat.com>]

Steve,

I'm a member of the OASIS technical committee that is working on both 
the addressing and management specifications.  I see my role there as 
tester/implementer.  Both specifications have a long way to go before 
they are usable IMO.  It is my plan that Dispatch will become compliant 
with both specifications, but it is a two-way street.  The TC needs also 
to respect the needs of implementers.

-Ted


On 10/11/2013 01:18 PM, Steve Huston wrote:
> Hi William,
>
> Thanks for expressing your view. And no reasonable person would discount it.
>
> Speaking for my own points, I would not argue that Dispatch (or any trailblazing development) should _wait_ for OASIS. However, since there are 2 areas of active work in OASIS that are related, and Dispatch is quickly gaining experience and backing, I would very much like to see Dispatch development folks actively involved in informing the OASIS technical committees of their experience and help to get the eventual standard(s) worked out in a way that's been proven in the field.
>
> -Steve
>
>> -----Original Message-----
>> From: William Henry [mailto:whenry@redhat.com]
>> Sent: Friday, October 11, 2013 12:49 PM
>> To: users@qpid.apache.org
>> Subject: Re: Qpid Dispatch Router component
>>
>>
>> I'd also add, because I'm sure that some of you wonder why is William
>> weighing in? What has he to do with AMQP/Qpid anyway besides being
>> historically linked to it?
>>
>> I've travelled globally talking about AMQP and Qpid to investment banks,
>> government agencies, telecoms, animation studios, stock exchanges etc. etc.
>> (strangers on planes ;-)  I'm sure others have done more to help promote
>> AMQP and Qpid but, despite my seemingly lack of, or invisible, community
>> involvement of late, I bet that list of others-that-have-done-more-to-
>> promote it is actually not a very large list. I can think of a few. I have other
>> technology areas I focus on but I continue to be involved in AMQP/Qpid
>> because I get pulled back in, and I still "believe". There is a massive
>> opportunity for AMQP that is more than just simple MOM.
>>
>> So my 2 cents at a high level:
>> 1) People love the AMQP story
>> 2) People love the AMQP network story - with a variety of AMQP based
>> assets.
>> 3) As we've introduced the concept of what Dispatch does, people love it!
>> Did I say love it?! And they do not see it as tangential to AMQP. But massively
>> complimentary. A Dispatch type network with (various) brokers backing it up
>> nearer to applications - that have different client interfaces (JMS, JCA,
>> Python, C/C++, Ruby etc.)
>> 4) The cloud will and is already playing a big part in AMQP adoption. We need
>> to capitalize.
>> 5) People want solutions now! Management now. Addressing nailed down
>> now.
>>
>> Best,
>> William
>>
>> ----- Original Message -----
>>> +1. Great post Gordon. Again something to be captured in Qpid
>>> +community
>>> pages/documentation.
>>>
>>> I'd also add we need to be innovating fast around AMQP now that we have
>> 1.0.
>>> We can't always wait on the OASIS process. It has taken a very long
>>> time to get here (not OASIS). There were some significant changes along
>> the way.
>>> Users are looking for solutions today.
>>>
>>> I understand there is a risk of getting ahead of ourselves in certain
>>> areas and having to backtrack. However people that have been investing
>>> in AMQP are looking for solutions to their problems ASAP and can't
>>> wait until every area has been worked out.
>>>
>>> I agree there is a risk of a high cost to "fixing" addressing and management.
>>> How can we get there faster? Surely innovations/projects like Dispatch
>>> can only help in driving the standardization faster as it and other
>>> artifacts hit issues.  I think Dispatch has already helped drive the
>>> addressing discussion.
>>>
>>> Let's press forward.
>>>
>>> William
>>>
>>> ----- Original Message -----
>>>> I don't claim to speak with any authority on Dispatch Router, but
>>>> fwiw, here's my answer to these two specific questions:
>>>>
>>>> On 10/10/2013 04:20 PM, Rob Godfrey wrote:
>>>>> How does a router differ from a broker?
>>>> I tend to have a more general view of what 'broker' might mean, but
>>>> in the context of statements that Dispatch Router 'is not a broker',
>>>> what is meant is the type of intermediary as exemplified by the two
>>>> existing Qpid brokers and others like them.
>>>>
>>>> As I see it, there are two key differences between Dispatch Router
>>>> and these.
>>>>
>>>> The first is that a principle of the Dispatch Router is that it does
>>>> not 'take ownership' of messages. This is in contrast to most
>>>> brokers where the publisher transfers responsibility for messages to
>>>> the broker and the broker then undertakes to deliver them allowing
>>>> the producer to forget about them. With Dispatch Router, the idea is
>>>> that it will route those messages, but any guarantee regarding
>>>> acceptance of those messages comes from the consumer(s) of the
>> message, not from the intermediary.
>>>> The second distinction is that Dispatch Router is from the start
>>>> designed to be a small piece in a distributed routing network of
>>>> interconnected components.
>>>>
>>>> Now, there is nothing that says a broker can't do these things. In
>>>> fact I added code a while back to the Qpid c++ broker that does the
>>>> former - relays messages through it, providing end-to-end guarantees.
>>>>
>>>> Likewise the distributed architecture has its roots in things like
>>>> the federation in the Qpid c++ broker (and similar solutions by other
>> brokers).
>>>> The Router however is setting out from the start to do *only* these
>>>> things. It is based on the idea that these are separate concerns
>>>> that can be split apart allowing 'brokers' to focus on queueing,
>>>> persistence, transactions, augmented functionality such as LVQ
>>>> patterns etc, and that the 'federation' logic can be provided in a
>>>> separate component that can hook up different brokers into a
>>>> federation as well as hooking in other types of AMQP based services
>>>> and allow clients to connect directly to the router as well.
>>>>
>>>> By focusing on one aspect its thought components can do a better job.
>>>> E.g. as I understand it, the Dispatch Router aims to address some of
>>>> the weaknesses of the federation support in qpidd, by providing for
>>>> redundant, fault-tolerant routes without duplication of messages,
>>>> adapting to failures and dynamically computing the best path between
>>>> two endpoints.
>>>>
>>>> I confess, for all its faults, federation has been one of my
>>>> favourite features of the c++ broker because of the potential I see
>>>> in it. That is why I started off adding some 1.0 related
>>>> improvements. However I think Ted's thinking on Dispatch Router is
>>>> quite compelling (though I also feel a lot of the necessary detail
>>>> is still missing and it still needs to be proven).
>>>>
>>>> For my part, I am excited to see how this develops, am keen to take
>>>> part if I can and would encourage anyone with an interest to chip in
>>>> with ideas, feedback, criticism, questions, code, use cases or whatever
>> else.
>>>> This began as Ted's 'brainchild' - and most things begin as
>>>> someone's idea - but it really is now open to anyone and everyone
>>>> who is interested. Get involved and shape the direction!
>>>>
>>>>> Would you expect any special client APIs to form part of the
>>>>> router package or not?
>>>> I'm not entirely sure if I understand the question, but there are
>>>> again two points I would make.
>>>>
>>>> First, in my opinion it is *vital* that no special client is
>>>> required to use Dispatch Router effectively. Transparency is a key
>>>> property. You should be able to point a JMS application or
>>>> qpid::messaging or any other compliant AMQP client at it. Any
>>>> special coupling between this and proton based clients would in my
>>>> view be a horrendous mistake. (Nothing like that is planned I'm
>>>> sure, I'm just using this as a hypothetical example to make the point).
>>>>
>>>> Second, the code in Dispatch Router is in theory designed around a
>>>> toolkit for building  AMQP 'containers' of different kinds, with the
>>>> router being one such example (another might be a proxy focused more
>>>> on enforcing ACLs at the edge). In theory this could be viewed as an
>>>> API of sorts. However I think at this point its better viewed as a
>>>> sensible desire for some internal structure and separation of
>>>> concerns. A publishable 'API' is in my view some way off and would
>>>> require a lot of work that would at this point distract from the
>>>> main goal, which is to define the behaviour of the router and implement
>> it.
>>>> --------------------------------------------------------------------
>>>> - To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
>>>> additional commands, e-mail: users-help@qpid.apache.org
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
>>> additional commands, e-mail: users-help@qpid.apache.org
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For additional
>> commands, e-mail: users-help@qpid.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

RE: Qpid Dispatch Router component

[Steve Huston <sh...@riverace.com>]

Hi William,

Thanks for expressing your view. And no reasonable person would discount it.

Speaking for my own points, I would not argue that Dispatch (or any trailblazing development) should _wait_ for OASIS. However, since there are 2 areas of active work in OASIS that are related, and Dispatch is quickly gaining experience and backing, I would very much like to see Dispatch development folks actively involved in informing the OASIS technical committees of their experience and help to get the eventual standard(s) worked out in a way that's been proven in the field.

-Steve

> -----Original Message-----
> From: William Henry [mailto:whenry@redhat.com]
> Sent: Friday, October 11, 2013 12:49 PM
> To: users@qpid.apache.org
> Subject: Re: Qpid Dispatch Router component
> 
> 
> I'd also add, because I'm sure that some of you wonder why is William
> weighing in? What has he to do with AMQP/Qpid anyway besides being
> historically linked to it?
> 
> I've travelled globally talking about AMQP and Qpid to investment banks,
> government agencies, telecoms, animation studios, stock exchanges etc. etc.
> (strangers on planes ;-)  I'm sure others have done more to help promote
> AMQP and Qpid but, despite my seemingly lack of, or invisible, community
> involvement of late, I bet that list of others-that-have-done-more-to-
> promote it is actually not a very large list. I can think of a few. I have other
> technology areas I focus on but I continue to be involved in AMQP/Qpid
> because I get pulled back in, and I still "believe". There is a massive
> opportunity for AMQP that is more than just simple MOM.
> 
> So my 2 cents at a high level:
> 1) People love the AMQP story
> 2) People love the AMQP network story - with a variety of AMQP based
> assets.
> 3) As we've introduced the concept of what Dispatch does, people love it!
> Did I say love it?! And they do not see it as tangential to AMQP. But massively
> complimentary. A Dispatch type network with (various) brokers backing it up
> nearer to applications - that have different client interfaces (JMS, JCA,
> Python, C/C++, Ruby etc.)
> 4) The cloud will and is already playing a big part in AMQP adoption. We need
> to capitalize.
> 5) People want solutions now! Management now. Addressing nailed down
> now.
> 
> Best,
> William
> 
> ----- Original Message -----
> > +1. Great post Gordon. Again something to be captured in Qpid
> > +community
> > pages/documentation.
> >
> > I'd also add we need to be innovating fast around AMQP now that we have
> 1.0.
> > We can't always wait on the OASIS process. It has taken a very long
> > time to get here (not OASIS). There were some significant changes along
> the way.
> > Users are looking for solutions today.
> >
> > I understand there is a risk of getting ahead of ourselves in certain
> > areas and having to backtrack. However people that have been investing
> > in AMQP are looking for solutions to their problems ASAP and can't
> > wait until every area has been worked out.
> >
> > I agree there is a risk of a high cost to "fixing" addressing and management.
> > How can we get there faster? Surely innovations/projects like Dispatch
> > can only help in driving the standardization faster as it and other
> > artifacts hit issues.  I think Dispatch has already helped drive the
> > addressing discussion.
> >
> > Let's press forward.
> >
> > William
> >
> > ----- Original Message -----
> > > I don't claim to speak with any authority on Dispatch Router, but
> > > fwiw, here's my answer to these two specific questions:
> > >
> > > On 10/10/2013 04:20 PM, Rob Godfrey wrote:
> > > > How does a router differ from a broker?
> > >
> > > I tend to have a more general view of what 'broker' might mean, but
> > > in the context of statements that Dispatch Router 'is not a broker',
> > > what is meant is the type of intermediary as exemplified by the two
> > > existing Qpid brokers and others like them.
> > >
> > > As I see it, there are two key differences between Dispatch Router
> > > and these.
> > >
> > > The first is that a principle of the Dispatch Router is that it does
> > > not 'take ownership' of messages. This is in contrast to most
> > > brokers where the publisher transfers responsibility for messages to
> > > the broker and the broker then undertakes to deliver them allowing
> > > the producer to forget about them. With Dispatch Router, the idea is
> > > that it will route those messages, but any guarantee regarding
> > > acceptance of those messages comes from the consumer(s) of the
> message, not from the intermediary.
> > >
> > > The second distinction is that Dispatch Router is from the start
> > > designed to be a small piece in a distributed routing network of
> > > interconnected components.
> > >
> > > Now, there is nothing that says a broker can't do these things. In
> > > fact I added code a while back to the Qpid c++ broker that does the
> > > former - relays messages through it, providing end-to-end guarantees.
> > >
> > > Likewise the distributed architecture has its roots in things like
> > > the federation in the Qpid c++ broker (and similar solutions by other
> brokers).
> > >
> > > The Router however is setting out from the start to do *only* these
> > > things. It is based on the idea that these are separate concerns
> > > that can be split apart allowing 'brokers' to focus on queueing,
> > > persistence, transactions, augmented functionality such as LVQ
> > > patterns etc, and that the 'federation' logic can be provided in a
> > > separate component that can hook up different brokers into a
> > > federation as well as hooking in other types of AMQP based services
> > > and allow clients to connect directly to the router as well.
> > >
> > > By focusing on one aspect its thought components can do a better job.
> > > E.g. as I understand it, the Dispatch Router aims to address some of
> > > the weaknesses of the federation support in qpidd, by providing for
> > > redundant, fault-tolerant routes without duplication of messages,
> > > adapting to failures and dynamically computing the best path between
> > > two endpoints.
> > >
> > > I confess, for all its faults, federation has been one of my
> > > favourite features of the c++ broker because of the potential I see
> > > in it. That is why I started off adding some 1.0 related
> > > improvements. However I think Ted's thinking on Dispatch Router is
> > > quite compelling (though I also feel a lot of the necessary detail
> > > is still missing and it still needs to be proven).
> > >
> > > For my part, I am excited to see how this develops, am keen to take
> > > part if I can and would encourage anyone with an interest to chip in
> > > with ideas, feedback, criticism, questions, code, use cases or whatever
> else.
> > > This began as Ted's 'brainchild' - and most things begin as
> > > someone's idea - but it really is now open to anyone and everyone
> > > who is interested. Get involved and shape the direction!
> > >
> > > > Would you expect any special client APIs to form part of the
> > > > router package or not?
> > >
> > > I'm not entirely sure if I understand the question, but there are
> > > again two points I would make.
> > >
> > > First, in my opinion it is *vital* that no special client is
> > > required to use Dispatch Router effectively. Transparency is a key
> > > property. You should be able to point a JMS application or
> > > qpid::messaging or any other compliant AMQP client at it. Any
> > > special coupling between this and proton based clients would in my
> > > view be a horrendous mistake. (Nothing like that is planned I'm
> > > sure, I'm just using this as a hypothetical example to make the point).
> > >
> > > Second, the code in Dispatch Router is in theory designed around a
> > > toolkit for building  AMQP 'containers' of different kinds, with the
> > > router being one such example (another might be a proxy focused more
> > > on enforcing ACLs at the edge). In theory this could be viewed as an
> > > API of sorts. However I think at this point its better viewed as a
> > > sensible desire for some internal structure and separation of
> > > concerns. A publishable 'API' is in my view some way off and would
> > > require a lot of work that would at this point distract from the
> > > main goal, which is to define the behaviour of the router and implement
> it.
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
> > > additional commands, e-mail: users-help@qpid.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
> > additional commands, e-mail: users-help@qpid.apache.org
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For additional
> commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[William Henry <wh...@redhat.com>]

I'd also add, because I'm sure that some of you wonder why is William weighing in? What has he to do with AMQP/Qpid anyway besides being historically linked to it?

I've travelled globally talking about AMQP and Qpid to investment banks, government agencies, telecoms, animation studios, stock exchanges etc. etc. (strangers on planes ;-)  I'm sure others have done more to help promote AMQP and Qpid but, despite my seemingly lack of, or invisible, community involvement of late, I bet that list of others-that-have-done-more-to-promote it is actually not a very large list. I can think of a few. I have other technology areas I focus on but I continue to be involved in AMQP/Qpid because I get pulled back in, and I still "believe". There is a massive opportunity for AMQP that is more than just simple MOM.  

So my 2 cents at a high level:
1) People love the AMQP story
2) People love the AMQP network story - with a variety of AMQP based assets.
3) As we've introduced the concept of what Dispatch does, people love it! Did I say love it?! And they do not see it as tangential to AMQP. But massively complimentary. A Dispatch type network with (various) brokers backing it up nearer to applications - that have different client interfaces (JMS, JCA, Python, C/C++, Ruby etc.)  
4) The cloud will and is already playing a big part in AMQP adoption. We need to capitalize. 
5) People want solutions now! Management now. Addressing nailed down now.   
 
Best,
William

----- Original Message -----
> +1. Great post Gordon. Again something to be captured in Qpid community
> pages/documentation.
> 
> I'd also add we need to be innovating fast around AMQP now that we have 1.0.
> We can't always wait on the OASIS process. It has taken a very long time to
> get here (not OASIS). There were some significant changes along the way.
> Users are looking for solutions today.
> 
> I understand there is a risk of getting ahead of ourselves in certain areas
> and having to backtrack. However people that have been investing in AMQP are
> looking for solutions to their problems ASAP and can't wait until every area
> has been worked out.
> 
> I agree there is a risk of a high cost to "fixing" addressing and management.
> How can we get there faster? Surely innovations/projects like Dispatch can
> only help in driving the standardization faster as it and other artifacts
> hit issues.  I think Dispatch has already helped drive the addressing
> discussion.
> 
> Let's press forward.
> 
> William
> 
> ----- Original Message -----
> > I don't claim to speak with any authority on Dispatch Router, but fwiw,
> > here's my answer to these two specific questions:
> > 
> > On 10/10/2013 04:20 PM, Rob Godfrey wrote:
> > > How does a router differ from a broker?
> > 
> > I tend to have a more general view of what 'broker' might mean, but in
> > the context of statements that Dispatch Router 'is not a broker', what
> > is meant is the type of intermediary as exemplified by the two existing
> > Qpid brokers and others like them.
> > 
> > As I see it, there are two key differences between Dispatch Router and
> > these.
> > 
> > The first is that a principle of the Dispatch Router is that it does not
> > 'take ownership' of messages. This is in contrast to most brokers where
> > the publisher transfers responsibility for messages to the broker and
> > the broker then undertakes to deliver them allowing the producer to
> > forget about them. With Dispatch Router, the idea is that it will route
> > those messages, but any guarantee regarding acceptance of those messages
> > comes from the consumer(s) of the message, not from the intermediary.
> > 
> > The second distinction is that Dispatch Router is from the start
> > designed to be a small piece in a distributed routing network of
> > interconnected components.
> > 
> > Now, there is nothing that says a broker can't do these things. In fact
> > I added code a while back to the Qpid c++ broker that does the former -
> > relays messages through it, providing end-to-end guarantees.
> > 
> > Likewise the distributed architecture has its roots in things like the
> > federation in the Qpid c++ broker (and similar solutions by other brokers).
> > 
> > The Router however is setting out from the start to do *only* these
> > things. It is based on the idea that these are separate concerns that
> > can be split apart allowing 'brokers' to focus on queueing, persistence,
> > transactions, augmented functionality such as LVQ patterns etc, and that
> > the 'federation' logic can be provided in a separate component that can
> > hook up different brokers into a federation as well as hooking in other
> > types of AMQP based services and allow clients to connect directly to
> > the router as well.
> > 
> > By focusing on one aspect its thought components can do a better job.
> > E.g. as I understand it, the Dispatch Router aims to address some of the
> > weaknesses of the federation support in qpidd, by providing for
> > redundant, fault-tolerant routes without duplication of messages,
> > adapting to failures and dynamically computing the best path between two
> > endpoints.
> > 
> > I confess, for all its faults, federation has been one of my favourite
> > features of the c++ broker because of the potential I see in it. That is
> > why I started off adding some 1.0 related improvements. However I think
> > Ted's thinking on Dispatch Router is quite compelling (though I also
> > feel a lot of the necessary detail is still missing and it still needs
> > to be proven).
> > 
> > For my part, I am excited to see how this develops, am keen to take part
> > if I can and would encourage anyone with an interest to chip in with
> > ideas, feedback, criticism, questions, code, use cases or whatever else.
> > This began as Ted's 'brainchild' - and most things begin as someone's
> > idea - but it really is now open to anyone and everyone who is
> > interested. Get involved and shape the direction!
> > 
> > > Would you expect any special client APIs to form part of the router
> > > package or not?
> > 
> > I'm not entirely sure if I understand the question, but there are again
> > two points I would make.
> > 
> > First, in my opinion it is *vital* that no special client is required to
> > use Dispatch Router effectively. Transparency is a key property. You
> > should be able to point a JMS application or qpid::messaging or any
> > other compliant AMQP client at it. Any special coupling between this and
> > proton based clients would in my view be a horrendous mistake. (Nothing
> > like that is planned I'm sure, I'm just using this as a hypothetical
> > example to make the point).
> > 
> > Second, the code in Dispatch Router is in theory designed around a
> > toolkit for building  AMQP 'containers' of different kinds, with the
> > router being one such example (another might be a proxy focused more on
> > enforcing ACLs at the edge). In theory this could be viewed as an API of
> > sorts. However I think at this point its better viewed as a sensible
> > desire for some internal structure and separation of concerns. A
> > publishable 'API' is in my view some way off and would require a lot of
> > work that would at this point distract from the main goal, which is to
> > define the behaviour of the router and implement it.
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> > 
> > 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[William Henry <wh...@redhat.com>]

+1. Great post Gordon. Again something to be captured in Qpid community pages/documentation.

I'd also add we need to be innovating fast around AMQP now that we have 1.0. We can't always wait on the OASIS process. It has taken a very long time to get here (not OASIS). There were some significant changes along the way. Users are looking for solutions today.

I understand there is a risk of getting ahead of ourselves in certain areas and having to backtrack. However people that have been investing in AMQP are looking for solutions to their problems ASAP and can't wait until every area has been worked out. 

I agree there is a risk of a high cost to "fixing" addressing and management. How can we get there faster? Surely innovations/projects like Dispatch can only help in driving the standardization faster as it and other artifacts hit issues.  I think Dispatch has already helped drive the addressing discussion.

Let's press forward.

William

----- Original Message -----
> I don't claim to speak with any authority on Dispatch Router, but fwiw,
> here's my answer to these two specific questions:
> 
> On 10/10/2013 04:20 PM, Rob Godfrey wrote:
> > How does a router differ from a broker?
> 
> I tend to have a more general view of what 'broker' might mean, but in
> the context of statements that Dispatch Router 'is not a broker', what
> is meant is the type of intermediary as exemplified by the two existing
> Qpid brokers and others like them.
> 
> As I see it, there are two key differences between Dispatch Router and
> these.
> 
> The first is that a principle of the Dispatch Router is that it does not
> 'take ownership' of messages. This is in contrast to most brokers where
> the publisher transfers responsibility for messages to the broker and
> the broker then undertakes to deliver them allowing the producer to
> forget about them. With Dispatch Router, the idea is that it will route
> those messages, but any guarantee regarding acceptance of those messages
> comes from the consumer(s) of the message, not from the intermediary.
> 
> The second distinction is that Dispatch Router is from the start
> designed to be a small piece in a distributed routing network of
> interconnected components.
> 
> Now, there is nothing that says a broker can't do these things. In fact
> I added code a while back to the Qpid c++ broker that does the former -
> relays messages through it, providing end-to-end guarantees.
> 
> Likewise the distributed architecture has its roots in things like the
> federation in the Qpid c++ broker (and similar solutions by other brokers).
> 
> The Router however is setting out from the start to do *only* these
> things. It is based on the idea that these are separate concerns that
> can be split apart allowing 'brokers' to focus on queueing, persistence,
> transactions, augmented functionality such as LVQ patterns etc, and that
> the 'federation' logic can be provided in a separate component that can
> hook up different brokers into a federation as well as hooking in other
> types of AMQP based services and allow clients to connect directly to
> the router as well.
> 
> By focusing on one aspect its thought components can do a better job.
> E.g. as I understand it, the Dispatch Router aims to address some of the
> weaknesses of the federation support in qpidd, by providing for
> redundant, fault-tolerant routes without duplication of messages,
> adapting to failures and dynamically computing the best path between two
> endpoints.
> 
> I confess, for all its faults, federation has been one of my favourite
> features of the c++ broker because of the potential I see in it. That is
> why I started off adding some 1.0 related improvements. However I think
> Ted's thinking on Dispatch Router is quite compelling (though I also
> feel a lot of the necessary detail is still missing and it still needs
> to be proven).
> 
> For my part, I am excited to see how this develops, am keen to take part
> if I can and would encourage anyone with an interest to chip in with
> ideas, feedback, criticism, questions, code, use cases or whatever else.
> This began as Ted's 'brainchild' - and most things begin as someone's
> idea - but it really is now open to anyone and everyone who is
> interested. Get involved and shape the direction!
> 
> > Would you expect any special client APIs to form part of the router
> > package or not?
> 
> I'm not entirely sure if I understand the question, but there are again
> two points I would make.
> 
> First, in my opinion it is *vital* that no special client is required to
> use Dispatch Router effectively. Transparency is a key property. You
> should be able to point a JMS application or qpid::messaging or any
> other compliant AMQP client at it. Any special coupling between this and
> proton based clients would in my view be a horrendous mistake. (Nothing
> like that is planned I'm sure, I'm just using this as a hypothetical
> example to make the point).
> 
> Second, the code in Dispatch Router is in theory designed around a
> toolkit for building  AMQP 'containers' of different kinds, with the
> router being one such example (another might be a proxy focused more on
> enforcing ACLs at the edge). In theory this could be viewed as an API of
> sorts. However I think at this point its better viewed as a sensible
> desire for some internal structure and separation of concerns. A
> publishable 'API' is in my view some way off and would require a lot of
> work that would at this point distract from the main goal, which is to
> define the behaviour of the router and implement it.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Ted Ross <tr...@redhat.com>]

On 10/11/2013 07:19 AM, Gordon Sim wrote:
> On 10/11/2013 11:52 AM, Rob Godfrey wrote:
>> On 11 October 2013 11:54, Gordon Sim <gs...@redhat.com> wrote:
>>
>>>
>>>
>>> Second, the code in Dispatch Router is in theory designed around a 
>>> toolkit
>>> for building  AMQP 'containers' of different kinds, with the router 
>>> being
>>> one such example (another might be a proxy focused more on enforcing 
>>> ACLs
>>> at the edge). In theory this could be viewed as an API of sorts. 
>>> However I
>>> think at this point its better viewed as a sensible desire for some
>>> internal structure and separation of concerns. A publishable 'API' 
>>> is in my
>>> view some way off and would require a lot of work that would at this 
>>> point
>>> distract from the main goal, which is to define the behaviour of the 
>>> router
>>> and implement it.
>>>
>>>
>> This would be really cool... though I would hope/think this would 
>> lead to a
>> spin-off new component for the toolkit rather than being part of 
>> Dispatch
>> itself?
>
> That would make sense to me, though I think we should cross that 
> bridge if and when we ever come to it. (E.g. if a new component 
> emerges that wants to reuse some of the same codebase).
>
>>  As such (and for the reasons you also allude to) I would think
>> this would not be a goal of Dispatch itself, but rather some sort of 
>> stated
>> aim and guiding principle of the development.
>
> To me, the focus has to be on demonstrating the vision (and proving it 
> works and has value). The rest is just good programming practice 
> (modular design, with minimal coupling and maximum cohesion).
>
>>  Going off-topic a bit I
>> guess... but would we see such a framework as having multiple
>> implementations (in different languages) or only in C?
>
> Again, to me the language used is secondary to the goal of proving the 
> concept with real, deployable software. I do think that the 
> communication patterns on top of AMQP between routers should be very 
> clearly spelled out to allow clones or alternative implementations 
> following the same rules to be developed if anyone anywhere has the 
> desire to do so.

+1 on all of the above.

I'll add one more motivation/criteria for Dispatch.  Performance is a 
primary goal.  I've gone to some lengths to avoid bottlenecks in 
multi-threaded memory management and buffer/field management.  I believe 
that if Qpid has a flexible and scalable interconnect that does not 
introduce performance slowdowns, there will be many very interesting 
things that we can layer over it.

-Ted

>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Fraser Adams <fr...@blueyonder.co.uk>]

On the topic of names and perhaps following the same track as particles 
with Proton, can I suggest "Boson" in honour of the forthcoming Nobel 
award to Professor Peter Higgs. It kind of seems appropriate on a couple 
of levels?

Frase

On 11/10/13 13:47, Gordon Sim wrote:
> On 10/11/2013 01:32 PM, Ted Ross wrote:
>> I have no love for the name.  But it has to be called *something* :)
>
> Indeed, and I reiterate how hard good naming is.
>
> Assuming 'Qpid Router' is undesirable as either making to big a claim 
> (there will never be any other Qpid thing that routes) or provides 
> insufficient 'identity', one option might be to follow the elementary 
> particles approach started with proton. How about 'gluon' or perhaps 
> 'graviton'? :-)
>
> I personally think a simple DX or QDX as an opaque name would be 
> better as it more obviously acknowledges that its merely an identifier 
> with little or no underlying description.
>
> Again, this should in no way imply any opposition to proceeding as is. 
> It is just an observation/suggestion.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Ted Ross <tr...@redhat.com>]

On 10/11/2013 08:47 AM, Gordon Sim wrote:
> On 10/11/2013 01:32 PM, Ted Ross wrote:
>> I have no love for the name.  But it has to be called *something* :)
>
> Indeed, and I reiterate how hard good naming is.
>
> Assuming 'Qpid Router' is undesirable as either making to big a claim 
> (there will never be any other Qpid thing that routes) or provides 
> insufficient 'identity', one option might be to follow the elementary 
> particles approach started with proton. How about 'gluon' or perhaps 
> 'graviton'? :-)
>
> I personally think a simple DX or QDX as an opaque name would be 
> better as it more obviously acknowledges that its merely an identifier 
> with little or no underlying description.

QDX is actually pretty good.  If we made that change, it wouldn't 
require significant file edits.

>
> Again, this should in no way imply any opposition to proceeding as is. 
> It is just an observation/suggestion.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Gordon Sim <gs...@redhat.com>]

On 10/11/2013 01:32 PM, Ted Ross wrote:
> I have no love for the name.  But it has to be called *something* :)

Indeed, and I reiterate how hard good naming is.

Assuming 'Qpid Router' is undesirable as either making to big a claim 
(there will never be any other Qpid thing that routes) or provides 
insufficient 'identity', one option might be to follow the elementary 
particles approach started with proton. How about 'gluon' or perhaps 
'graviton'? :-)

I personally think a simple DX or QDX as an opaque name would be better 
as it more obviously acknowledges that its merely an identifier with 
little or no underlying description.

Again, this should in no way imply any opposition to proceeding as is. 
It is just an observation/suggestion.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Ted Ross <tr...@redhat.com>]

I have no love for the name.  But it has to be called *something* :)

On 10/11/2013 07:29 AM, Gordon Sim wrote:
> One other point, I do think 'Dispatch' is a poor name. It is neither 
> distinctive nor descriptive.
>
> Appending 'Router' to it only underlines this in my view. That at 
> least begins to explain what the component does. However 'Dispatch' 
> then adds very little else.
>
> That said I know how hard naming can be and this isn't an objection to 
> making progress, simply an observation.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Gordon Sim <gs...@redhat.com>]

One other point, I do think 'Dispatch' is a poor name. It is neither 
distinctive nor descriptive.

Appending 'Router' to it only underlines this in my view. That at least 
begins to explain what the component does. However 'Dispatch' then adds 
very little else.

That said I know how hard naming can be and this isn't an objection to 
making progress, simply an observation.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Rob Godfrey <ro...@gmail.com>]

On 11 October 2013 13:19, Gordon Sim <gs...@redhat.com> wrote:

> On 10/11/2013 11:52 AM, Rob Godfrey wrote:
>
>> On 11 October 2013 11:54, Gordon Sim <gs...@redhat.com> wrote:
>>
>>
>>>
>>> Second, the code in Dispatch Router is in theory designed around a
>>> toolkit
>>> for building  AMQP 'containers' of different kinds, with the router being
>>> one such example (another might be a proxy focused more on enforcing ACLs
>>> at the edge). In theory this could be viewed as an API of sorts. However
>>> I
>>> think at this point its better viewed as a sensible desire for some
>>> internal structure and separation of concerns. A publishable 'API' is in
>>> my
>>> view some way off and would require a lot of work that would at this
>>> point
>>> distract from the main goal, which is to define the behaviour of the
>>> router
>>> and implement it.
>>>
>>>
>>>  This would be really cool... though I would hope/think this would lead
>> to a
>> spin-off new component for the toolkit rather than being part of Dispatch
>> itself?
>>
>
> That would make sense to me, though I think we should cross that bridge if
> and when we ever come to it. (E.g. if a new component emerges that wants to
> reuse some of the same codebase).
>
>
Yeah - that makes sense.


>
>   As such (and for the reasons you also allude to) I would think
>> this would not be a goal of Dispatch itself, but rather some sort of
>> stated
>> aim and guiding principle of the development.
>>
>
> To me, the focus has to be on demonstrating the vision (and proving it
> works and has value). The rest is just good programming practice (modular
> design, with minimal coupling and maximum cohesion).
>
>
+1


>
>   Going off-topic a bit I
>> guess... but would we see such a framework as having multiple
>> implementations (in different languages) or only in C?
>>
>
> Again, to me the language used is secondary to the goal of proving the
> concept with real, deployable software. I do think that the communication
> patterns on top of AMQP between routers should be very clearly spelled out
> to allow clones or alternative implementations following the same rules to
> be developed if anyone anywhere has the desire to do so.
>
>
>
Agreed on all the above :)

-- Rob


>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: Qpid Dispatch Router component

[Gordon Sim <gs...@redhat.com>]

On 10/11/2013 11:52 AM, Rob Godfrey wrote:
> On 11 October 2013 11:54, Gordon Sim <gs...@redhat.com> wrote:
>
>>
>>
>> Second, the code in Dispatch Router is in theory designed around a toolkit
>> for building  AMQP 'containers' of different kinds, with the router being
>> one such example (another might be a proxy focused more on enforcing ACLs
>> at the edge). In theory this could be viewed as an API of sorts. However I
>> think at this point its better viewed as a sensible desire for some
>> internal structure and separation of concerns. A publishable 'API' is in my
>> view some way off and would require a lot of work that would at this point
>> distract from the main goal, which is to define the behaviour of the router
>> and implement it.
>>
>>
> This would be really cool... though I would hope/think this would lead to a
> spin-off new component for the toolkit rather than being part of Dispatch
> itself?

That would make sense to me, though I think we should cross that bridge 
if and when we ever come to it. (E.g. if a new component emerges that 
wants to reuse some of the same codebase).

>  As such (and for the reasons you also allude to) I would think
> this would not be a goal of Dispatch itself, but rather some sort of stated
> aim and guiding principle of the development.

To me, the focus has to be on demonstrating the vision (and proving it 
works and has value). The rest is just good programming practice 
(modular design, with minimal coupling and maximum cohesion).

>  Going off-topic a bit I
> guess... but would we see such a framework as having multiple
> implementations (in different languages) or only in C?

Again, to me the language used is secondary to the goal of proving the 
concept with real, deployable software. I do think that the 
communication patterns on top of AMQP between routers should be very 
clearly spelled out to allow clones or alternative implementations 
following the same rules to be developed if anyone anywhere has the 
desire to do so.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Rob Godfrey <ro...@gmail.com>]

On 11 October 2013 11:54, Gordon Sim <gs...@redhat.com> wrote:

>
>
> Second, the code in Dispatch Router is in theory designed around a toolkit
> for building  AMQP 'containers' of different kinds, with the router being
> one such example (another might be a proxy focused more on enforcing ACLs
> at the edge). In theory this could be viewed as an API of sorts. However I
> think at this point its better viewed as a sensible desire for some
> internal structure and separation of concerns. A publishable 'API' is in my
> view some way off and would require a lot of work that would at this point
> distract from the main goal, which is to define the behaviour of the router
> and implement it.
>
>
This would be really cool... though I would hope/think this would lead to a
spin-off new component for the toolkit rather than being part of Dispatch
itself?  As such (and for the reasons you also allude to) I would think
this would not be a goal of Dispatch itself, but rather some sort of stated
aim and guiding principle of the development.  Going off-topic a bit I
guess... but would we see such a framework as having multiple
implementations (in different languages) or only in C?

-- Rob


>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: Qpid Dispatch Router component

[Gordon Sim <gs...@redhat.com>]

I don't claim to speak with any authority on Dispatch Router, but fwiw,
here's my answer to these two specific questions:

On 10/10/2013 04:20 PM, Rob Godfrey wrote:
> How does a router differ from a broker?

I tend to have a more general view of what 'broker' might mean, but in 
the context of statements that Dispatch Router 'is not a broker', what 
is meant is the type of intermediary as exemplified by the two existing 
Qpid brokers and others like them.

As I see it, there are two key differences between Dispatch Router and 
these.

The first is that a principle of the Dispatch Router is that it does not 
'take ownership' of messages. This is in contrast to most brokers where 
the publisher transfers responsibility for messages to the broker and 
the broker then undertakes to deliver them allowing the producer to 
forget about them. With Dispatch Router, the idea is that it will route 
those messages, but any guarantee regarding acceptance of those messages 
comes from the consumer(s) of the message, not from the intermediary.

The second distinction is that Dispatch Router is from the start 
designed to be a small piece in a distributed routing network of 
interconnected components.

Now, there is nothing that says a broker can't do these things. In fact 
I added code a while back to the Qpid c++ broker that does the former - 
relays messages through it, providing end-to-end guarantees.

Likewise the distributed architecture has its roots in things like the 
federation in the Qpid c++ broker (and similar solutions by other brokers).

The Router however is setting out from the start to do *only* these 
things. It is based on the idea that these are separate concerns that 
can be split apart allowing 'brokers' to focus on queueing, persistence, 
transactions, augmented functionality such as LVQ patterns etc, and that 
the 'federation' logic can be provided in a separate component that can 
hook up different brokers into a federation as well as hooking in other 
types of AMQP based services and allow clients to connect directly to 
the router as well.

By focusing on one aspect its thought components can do a better job. 
E.g. as I understand it, the Dispatch Router aims to address some of the 
weaknesses of the federation support in qpidd, by providing for 
redundant, fault-tolerant routes without duplication of messages, 
adapting to failures and dynamically computing the best path between two 
endpoints.

I confess, for all its faults, federation has been one of my favourite 
features of the c++ broker because of the potential I see in it. That is 
why I started off adding some 1.0 related improvements. However I think 
Ted's thinking on Dispatch Router is quite compelling (though I also 
feel a lot of the necessary detail is still missing and it still needs 
to be proven).

For my part, I am excited to see how this develops, am keen to take part 
if I can and would encourage anyone with an interest to chip in with 
ideas, feedback, criticism, questions, code, use cases or whatever else. 
This began as Ted's 'brainchild' - and most things begin as someone's 
idea - but it really is now open to anyone and everyone who is 
interested. Get involved and shape the direction!

> Would you expect any special client APIs to form part of the router
> package or not?

I'm not entirely sure if I understand the question, but there are again 
two points I would make.

First, in my opinion it is *vital* that no special client is required to 
use Dispatch Router effectively. Transparency is a key property. You 
should be able to point a JMS application or qpid::messaging or any 
other compliant AMQP client at it. Any special coupling between this and 
proton based clients would in my view be a horrendous mistake. (Nothing 
like that is planned I'm sure, I'm just using this as a hypothetical 
example to make the point).

Second, the code in Dispatch Router is in theory designed around a 
toolkit for building  AMQP 'containers' of different kinds, with the 
router being one such example (another might be a proxy focused more on 
enforcing ACLs at the edge). In theory this could be viewed as an API of 
sorts. However I think at this point its better viewed as a sensible 
desire for some internal structure and separation of concerns. A 
publishable 'API' is in my view some way off and would require a lot of 
work that would at this point distract from the main goal, which is to 
define the behaviour of the router and implement it.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Gordon Sim <gs...@redhat.com>]

It was pointed out to me that sloppy wording on my part may have led to 
an impression other than the one intended. At the risk of digging a 
bigger hole, I'd like to try to correct that.

On 10/11/2013 12:36 PM, Gordon Sim wrote:
> As I said before, my interest is not in reforming OASIS. My interest is
> in the Qpid project and in demonstrable interoperability and choice.

The statement above was simply intended to clarify the purpose of my 
comments in this thread.

I was trying to ensure the conversation did not veer off in an 
unintended direction rather than voicing any more general opinion.

I have an interest in *anything* that can deliver practical 
interoperability and choice for messaging users.

I apologise for the lack of precision in my original email and any 
misunderstanding that may have caused.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Gordon Sim <gs...@redhat.com>]

On 10/11/2013 12:27 AM, Steve Huston wrote:
> The argument for evolving "de facto" standards is not really
> pertinent here (in the context of addressing and management). De
> facto standards emerge when some product/idea is developed and turned
> loose and people take off and run with it. In this case, work is
> ongoing at OASIS and other products are (I assume) implementing
> them.

One of the biggest lessons I've learned during the years of AMQP's 
painful evolution, is that with standards, the technical details are not 
the only important thing. Indeed often they aren't even the most 
important thing. For want of a better word, 'social' aspects are critical.

For me, this is the area where AMQP made the biggest mistakes. Now that 
1.0 is here and is becoming a reality, I really hoped the same mistakes 
would not be perpetuated. AMQP has a history of mandating things that 
no-one implements or that are implemented grudgingly. It has a history 
of willing participants feeling excluded. (The dearth of practical 
choice for 1.0 clients is still a very real problem).

As I said before, my interest is not in reforming OASIS. My interest is 
in the Qpid project and in demonstrable interoperability and choice. I 
believe open source works best when software is produced by developers 
who want to do it, who feel personally engaged in and excited about what 
they are implementing, not when they feel coerced by outside forces.

The AMQP core specification is here and I fully accept its authority. 
The only 'authority' I recognise beyond that is the Qpid community and 
evidence of consensus with other products, particularly other open 
source projects.

A key principle at Apache is the 'faithful implementation of standards'. 
I firmly believe in that. The standards we do adopt we should implement 
faithfully. But simply because Qpid was founded to implement and promote 
AMQP does not mean that any additional 'emerging standard' being worked 
on under that name has special authority over the space it stakes out 
for itself.

That is not to say that I think they should be ignored. If there is no 
good reason not to adopt them, doing so makes sense. However I 
personally do not accept the notion that I have any obligation to engage 
with the OASIS process to address whatever issues I may have. My 
obligation is to discuss my views first and foremost with the Qpid 
community and the users of whatever software I am working on. If the 
view of the community is that only the 'emerging standard' can be 
implemented, so be it. It is the authority of the community I would 
respect there, not that of OASIS.

I stress again that I *want* genuine interoperability and real choice 
for users. I'm just not prepared to restrict myself to an approach and 
process I have lost faith in.

This is my own *personal* viewpoint and does not in anyway reflect the 
views of my employer or that of any colleague, friend or associate!

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

RE: Qpid Dispatch Router component

[Steve Huston <sh...@riverace.com>]

Hi guys,

> -----Original Message-----
> From: Rob Godfrey [mailto:rob.j.godfrey@gmail.com]
> Sent: Thursday, October 10, 2013 11:21 AM
> To: users@qpid.apache.org
> Subject: Re: Qpid Dispatch Router component
> 
> On 10 October 2013 15:35, Ted Ross <tr...@redhat.com> wrote:
> 
> >
> > On 10/10/2013 04:38 AM, Rob Godfrey wrote:
> >
> >> My main concern is that I believe that Qpid should be primarily
> >> directed at implementing AMQP standards, and building resuable
> >> toolkits and components that fit into any AMQP network. I'd be very
> >> concerned if we were inventing alternative management protocols, or
> >> building components that only interoperated with other Qpid tools.
> >> So, personally I'd like to see statement around components saying
> >> that they will be fully implementing emerging AMQP Management,
> AMQP
> >> addressing, etc. standards, and that we as a project then ensure we stick
> to these goals.
> >>
> >
> > Rob,
> >
> > Here's where I'm going to have to disagree with you in principle. I
> > believe that Qpid should be primarily directed at innovating with AMQP
> > and helping to drive the AMQP standards where appropriate.  If Qpid
> > doesn't do it, somebody else will.  I should point out that almost
> > everything we do here is well ahead of the standards, including the JMS
> client.
> >
> > The thing I object to in your statement is the direction of flow from
> > the standard to the implementation.  Standards bodies _do not
> > innovate_.  If the emerging standards are such that a particular
> > valuable innovation cannot be done, the standard needs to change or be
> > ignored.  Qpid must not allow itself to be put in the position of
> > meekly sitting and waiting for the tablets to come from on high before
> implementing.
> >
> >
> I'm not saying that there is a direction of standard -> implementation, but
> that if there is a standard we should be implementing it rather than rolling
> our own which conflicts or substantially overlaps.  If we see a standard
> emerging at OASIS and do not believe it is correct then we should be working
> to fix it at that venue.  If we do work which we think is generally applicable to
> other AMQP implementations then we should be considering whether we
> want to push it as a standard at OASiS or elsewhere.

The above is, as I see it, the crux of the matter. I agree with Rob on this item. Addressing and management are evolving standards in OASIS and they should not be ignored. The argument for evolving "de facto" standards is not really pertinent here (in the context of addressing and management). De facto standards emerge when some product/idea is developed and turned loose and people take off and run with it. In this case, work is ongoing at OASIS and other products are (I assume) implementing them. Ignoring that will only invite difficulties down the road, especially if Dispatch ends up only able to talk to itself. And I REALLY, REALLY want Dispatch to take off - I have big ideas for its use already.

As far as the idea of a AMQP router, now that's an opportunity for de facto standard(s). As long as it interoperates with any other AMQP 1.0 endpoint that uses AMQP standard addressing and it responds correctly to AMQP standard management.

Picture yourself as in Cisco's shoes 25-30 years ago. Take it from that approach and you'll be golden.

> I absolutely don't think that Qpid should be restricted in scope to simply
> implementing the standards, and I strongly believe that Qpid can be a place
> where we innovate and then work towards bringing behaviours to a
> standards body.  However I also think that when we do so we should state
> up front what it is we are trying to achieve.  I also don't think that qpid should
> become a mini-GitHub for any project that is tangentially related to AMQP to
> simply use as a code repository.
> 
> So, in the case of Dispatch, it certainly seems to include innovative ideas
> which do not form part of any AMQP standard (proposed or current) but also
> seems to hint at overlaps with such (emerging) standards in addressing and
> in management.  In addressing I think it's important that any requirements
> for addressing that you believe are important for Dispatch are discussed and
> considered in the OASIS AMQP work... in the Management space I would
> very much hope that the emerging Management spec for AMQP would
> prove to be a foundation on which functionality could be built and I would be
> concerned for a number of reasons if you felt that Dispatch
> couldn't/shouldn't be using them.

Absolutely.

-Steve


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Qpid Dispatch Router component

[Rob Godfrey <ro...@gmail.com>]

On 10 October 2013 15:35, Ted Ross <tr...@redhat.com> wrote:

>
> On 10/10/2013 04:38 AM, Rob Godfrey wrote:
>
>> My main concern is that I believe that Qpid should be primarily directed
>> at implementing AMQP standards, and building resuable toolkits and
>> components that fit into any AMQP network. I'd be very concerned if we were
>> inventing alternative management protocols, or building components that
>> only interoperated with other Qpid tools. So, personally I'd like to see
>> statement around components saying that they will be fully implementing
>> emerging AMQP Management, AMQP addressing, etc. standards, and that we as a
>> project then ensure we stick to these goals.
>>
>
> Rob,
>
> Here's where I'm going to have to disagree with you in principle. I
> believe that Qpid should be primarily directed at innovating with AMQP and
> helping to drive the AMQP standards where appropriate.  If Qpid doesn't do
> it, somebody else will.  I should point out that almost everything we do
> here is well ahead of the standards, including the JMS client.
>
> The thing I object to in your statement is the direction of flow from the
> standard to the implementation.  Standards bodies _do not innovate_.  If
> the emerging standards are such that a particular valuable innovation
> cannot be done, the standard needs to change or be ignored.  Qpid must not
> allow itself to be put in the position of meekly sitting and waiting for
> the tablets to come from on high before implementing.
>
>
I'm not saying that there is a direction of standard -> implementation, but
that if there is a standard we should be implementing it rather than
rolling our own which conflicts or substantially overlaps.  If we see a
standard emerging at OASIS and do not believe it is correct then we should
be working to fix it at that venue.  If we do work which we think is
generally applicable to other AMQP implementations then we should be
considering whether we want to push it as a standard at OASiS or elsewhere.

I absolutely don't think that Qpid should be restricted in scope to simply
implementing the standards, and I strongly believe that Qpid can be a place
where we innovate and then work towards bringing behaviours to a standards
body.  However I also think that when we do so we should state up front
what it is we are trying to achieve.  I also don't think that qpid should
become a mini-GitHub for any project that is tangentially related to AMQP
to simply use as a code repository.

So, in the case of Dispatch, it certainly seems to include innovative ideas
which do not form part of any AMQP standard (proposed or current) but also
seems to hint at overlaps with such (emerging) standards in addressing and
in management.  In addressing I think it's important that any requirements
for addressing that you believe are important for Dispatch are discussed
and considered in the OASIS AMQP work... in the Management space I would
very much hope that the emerging Management spec for AMQP would prove to be
a foundation on which functionality could be built and I would be concerned
for a number of reasons if you felt that Dispatch couldn't/shouldn't be
using them.


> So here's my proposed statement regarding Dispatch:
>
> Qpid Dispatch is an implementation of an AMQP-compliant router. Dispatch
> is pursuing a specific approach to routing and addressing that may differ
> from other approaches.  The implementation will conform to all relevant
> emerging standards (Management, Addressing, and Security) to the extent
> that it is practical.  In the event that there are parts of the
> specifications that are not practical to implement, we shall provide
> specifics to the standards committee in an effort to improve the
> specifications.
>
>
Around AMQP compliance that seems reasonable, but from the rest of that
statement I'm not sure what "AMQP-compliant router" means... How does a
router differ from a broker?  Would you expect any special client APIs to
form part of the router package or not?  Would you expect any other
implementations of Dispatch, or is it intended only to be in C?  Is it
intended to be embedded in other programs or always to act as a stand-alone
executable?

What I'm getting at with a scope statement is that we should be clear at
the outset of projects what we are looking to achieve.  We may deliberately
leave thing very open, but then we should state that clearly too.

In order to be able to have a clear story about how the components of Qpid
fit together I think we should be clear before we set out on the journey of
a component what we are looking to do (and what we are explicitly not
looking to do).  For previous components I don't think we've done that well
enough (and for the upcoming JMS client I want to go back and rectify that
omission).

-- Rob

-Ted
>
>
> ------------------------------**------------------------------**---------
>
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: Qpid Dispatch Router component

[Ted Ross <tr...@redhat.com>]

On 10/10/2013 04:38 AM, Rob Godfrey wrote:
> My main concern is that I believe that Qpid should be primarily 
> directed at implementing AMQP standards, and building resuable 
> toolkits and components that fit into any AMQP network. I'd be very 
> concerned if we were inventing alternative management protocols, or 
> building components that only interoperated with other Qpid tools. So, 
> personally I'd like to see statement around components saying that 
> they will be fully implementing emerging AMQP Management, AMQP 
> addressing, etc. standards, and that we as a project then ensure we 
> stick to these goals.

Rob,

Here's where I'm going to have to disagree with you in principle. I 
believe that Qpid should be primarily directed at innovating with AMQP 
and helping to drive the AMQP standards where appropriate.  If Qpid 
doesn't do it, somebody else will.  I should point out that almost 
everything we do here is well ahead of the standards, including the JMS 
client.

The thing I object to in your statement is the direction of flow from 
the standard to the implementation.  Standards bodies _do not 
innovate_.  If the emerging standards are such that a particular 
valuable innovation cannot be done, the standard needs to change or be 
ignored.  Qpid must not allow itself to be put in the position of meekly 
sitting and waiting for the tablets to come from on high before 
implementing.

So here's my proposed statement regarding Dispatch:

Qpid Dispatch is an implementation of an AMQP-compliant router. Dispatch 
is pursuing a specific approach to routing and addressing that may 
differ from other approaches.  The implementation will conform to all 
relevant emerging standards (Management, Addressing, and Security) to 
the extent that it is practical.  In the event that there are parts of 
the specifications that are not practical to implement, we shall provide 
specifics to the standards committee in an effort to improve the 
specifications.

-Ted


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org