Qpid and AMQP 1-0: Plans?

[Robert Godfrey <ro...@gmail.com>]

All,

As we are now reaching the end of the 0.10 release process, I think it's
probably an opportune moment to look at the goals of the Qpid project for
the rest of 2011 and beyond.

As many of you are no doubt aware, I have been spending the majority of my
time over the last years on the AMQP 1-0 standard, which we can shortly
expect to be released in its final form. In advance of this I think we need
to be planning on how we are going to adapt Qpid so that AMQP 1-0 becomes
its primary protocol.

What are people's views about how we approach this?

Cheers,
Rob

Re: Qpid and AMQP 1-0: Plans?

[Robert Godfrey <ro...@gmail.com>]

On 24 March 2011 16:19, Rafael Schloming <ra...@redhat.com> wrote:

> On 03/24/2011 11:07 AM, Robert Godfrey wrote:
>
>> On 24 March 2011 15:56, Rafael Schloming<ra...@redhat.com>  wrote:
>>
>
>
*snip*


>  On 03/24/2011 10:43 AM, Robert Godfrey wrote: rest of qpid...
>>>
>>
>>
*snip*


> Branch is probably the wrong name here... but I'm thinking that this
>> doesn't
>> really fit under the same umbrella as the rest of the components which
>> have
>> a single release cycle currently.  What we possibly want is something that
>> is in parallel to the existing codebase... repos/asf/qpid/trunk/transport
>> in
>> parallel to repos/asf/qpid/trunk/qpid would be one way of doing it...
>> Although looking at other Apache projects it seems like the common way of
>> doing this is
>>
>> .../asf/<project>/<component>/trunk or even
>> .../asf/<project>/<component>/<sub-component>/trunk (see ant, commons,
>> httpd, etc.)
>>
>> Though this would obviously require a little more re-organisation for us
>> all...
>>
>
> I agree we want something parallel. I'm not particularly bothered which
> way, I probably have a mild preference for the <project>/<component>/trunk,
> but if such a reorg would prove to be a pain for some reason I wouldn't be
> particularly bothered by the other way.
>
>
Yeah - that would be my preference too - and also in line with the other
Apache projects.

Given the sort of deadlines we're looking at - ApacheCon, and also the AMQP
1-0 published timelines - I think we need to make a start on this as soon as
possible.  Perhaps after 0.10 is release we could look at putting the svn
change (bringing us in line with other Apache projects) to a vote?

I think getting the UML diagrams in place would be tremendously helpful for
me (at least) to get a slightly better field for where you think the
boundaries between the component parts of the transport layer lie.  The
sooner we have a place for all this in our repo, the better.

At some point I want to step back a bit too, and think how we might support
existing AMQP protocols.  Certainly from the Java side we have endeavoured
to try to support *all* versions of AMQP and translate between them.  In
line with my vision of Qpid as being the de facto reference implementation I
do think we should strive to maintain this aspect of supporting historical
versions somehow too...  It needn't be using the same transport layer, but
it would be much neater if we could :-)

-- Rob


>
> --Rafael
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>

Re: Qpid and AMQP 1-0: Plans?

[Rafael Schloming <ra...@redhat.com>]

On 03/24/2011 11:07 AM, Robert Godfrey wrote:
> On 24 March 2011 15:56, Rafael Schloming<ra...@redhat.com>  wrote:
>
>> On 03/24/2011 10:43 AM, Robert Godfrey wrote:
>>
>>> So - obviously we need to hear a few more voices on this... But personally
>>> I'm *really* keen to get the project moving towards AMQP 1-0 ASAP.  In my
>>> spare time I've sort of hacked up an AMQP 1-0 Java client, and then
>>> spliced
>>> that onto the Qpid Java Broker...  It's not code I would really consider
>>> putting into Qpid, but this (and the code you've got on github) may be
>>> able
>>> to help us bootstrap testing our 1-0 efforts.
>>>
>>> If we follow the idea of the common transport, I would guess that the best
>>> way to progress this is probably to create a branch where we can start out
>>> defining and implementing this transport as a standalone entity.  It
>>> doesn;t
>>> seem to make sense to me to have this in trunk and tie it into the release
>>> cycles of the current Qpid work...
>>>
>>> Thoughts?
>>>
>>
>> For developing the transport itself, I'm not sure it matters much whether
>> we branch or not if we're treating it as an independent component. Obviously
>> if we start integrating it into other pieces then a branch for those pieces
>> would be called for. The release cycle question is certainly interesting. I
>> agree it shouldn't really be tied to the trunk release in one sense, but I
>> do think it would be good to have something ready for this year's ApacheCon,
>> and given that it might make sense to have deliverable milestones that
>> coincide with the trunk releases purely as a means of focusing progress.
>>
>>
>  From a theoretical point of view I agree.
>
> Logically we are starting from a clean slate and developing what amounts to
> an independent library which we will then, hopefully, make into a dependency
> of the rest of qpid...
>
> Branch is probably the wrong name here... but I'm thinking that this doesn't
> really fit under the same umbrella as the rest of the components which have
> a single release cycle currently.  What we possibly want is something that
> is in parallel to the existing codebase... repos/asf/qpid/trunk/transport in
> parallel to repos/asf/qpid/trunk/qpid would be one way of doing it...
> Although looking at other Apache projects it seems like the common way of
> doing this is
>
> .../asf/<project>/<component>/trunk or even
> .../asf/<project>/<component>/<sub-component>/trunk (see ant, commons,
> httpd, etc.)
>
> Though this would obviously require a little more re-organisation for us
> all...

I agree we want something parallel. I'm not particularly bothered which 
way, I probably have a mild preference for the 
<project>/<component>/trunk, but if such a reorg would prove to be a 
pain for some reason I wouldn't be particularly bothered by the other way.

--Rafael

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Qpid and AMQP 1-0: Plans?

[Justin Ross <jr...@redhat.com>]

On Thu, 24 Mar 2011, Andrew Stitcher wrote:

> On Thu, 2011-03-24 at 16:07 +0100, Robert Godfrey wrote:
>> ...
>> Branch is probably the wrong name here... but I'm thinking that this doesn't
>> really fit under the same umbrella as the rest of the components which have
>> a single release cycle currently.  What we possibly want is something that
>> is in parallel to the existing codebase... repos/asf/qpid/trunk/transport in
>> parallel to repos/asf/qpid/trunk/qpid would be one way of doing it...
>> Although looking at other Apache projects it seems like the common way of
>> doing this is
>>
>> .../asf/<project>/<component>/trunk or even
>> .../asf/<project>/<component>/<sub-component>/trunk (see ant, commons,
>> httpd, etc.)
>>
>> Though this would obviously require a little more re-organisation for us
>> all...
>
> Actually not necessarily, we already have a superfluous "qpid" directory
> at the top level of trunk, so we could introduce "amqp-transport" at
> that level if we wanted to.
>
> ie .../asf/qpid/trunk/amqp-transport/...
>
> I'm not necessarily advocating this, but it would avoid a whole lot of
> top level moves that might heavily confuse the git mirroring process.
>
> [BTW sorry to spoil the double act]

(Haha!)

Looking further into the future, I wonder if we shouldn't plan not just 
for common infrastructure but for superstructure, too:

   qpid/trunk/qpid
   qpid/trunk/qpid/cpp
   qpid/trunk/qpid/java
   (etc.)

   vs.

   qpid/trunk/floobarb
   qpid/trunk/floobarb/common
   qpid/trunk/floobarb/transport
   qpid/trunk/floobarb/client
   qpid/trunk/floobarb/broker

One of my goals is to have a common entry point for building and testing, 
and that's made more difficult by spinning one of the components off.

I'd like whatever the new plan is to set us on the path to "one project, 
for reals".

Justin

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Qpid and AMQP 1-0: Plans?

[Robert Godfrey <ro...@gmail.com>]

On 24 March 2011 16:55, Andrew Stitcher <as...@redhat.com> wrote:

> On Thu, 2011-03-24 at 16:07 +0100, Robert Godfrey wrote:
> > ...
> > Branch is probably the wrong name here... but I'm thinking that this
> doesn't
> > really fit under the same umbrella as the rest of the components which
> have
> > a single release cycle currently.  What we possibly want is something
> that
> > is in parallel to the existing codebase... repos/asf/qpid/trunk/transport
> in
> > parallel to repos/asf/qpid/trunk/qpid would be one way of doing it...
> > Although looking at other Apache projects it seems like the common way of
> > doing this is
> >
> > .../asf/<project>/<component>/trunk or even
> > .../asf/<project>/<component>/<sub-component>/trunk (see ant, commons,
> > httpd, etc.)
> >
> > Though this would obviously require a little more re-organisation for us
> > all...
>
> Actually not necessarily, we already have a superfluous "qpid" directory
> at the top level of trunk, so we could introduce "amqp-transport" at
> that level if we wanted to.
>
> ie .../asf/qpid/trunk/amqp-transport/...
>
> I'm not necessarily advocating this, but it would avoid a whole lot of
> top level moves that might heavily confuse the git mirroring process.
>
> [BTW sorry to spoil the double act]
>
>
yeah - we don't really *need* to fix the rest of qpid at the same time as we
start on using a sensible structure in one place... but symmetry and
aesthetics kind of compel me to suggest it :-)

Having seen how every other project does it, I think

qpid/<component>/trunk is probably the right way to go ...

We can leave the existing code in qpid/trunk/qpid if we like ... it won't
clash... it's just (and has always been) yuck!

-- Rob


> Andrew
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>

RE: Qpid and AMQP 1-0: Plans?

[Steve Huston <sh...@riverace.com>]

FWIW, I concur with Rob's assessment of strategy and layering.

-Steve

> -----Original Message-----
> From: Robert Godfrey [mailto:rob.j.godfrey@gmail.com] 
> Sent: Friday, March 25, 2011 9:18 AM
> To: dev@qpid.apache.org; cctrieloff@redhat.com
> Subject: Re: Qpid and AMQP 1-0: Plans?
> 
> 
> On 25 March 2011 13:44, Carl Trieloff <cc...@redhat.com> wrote:
> 
> >
> >
> > It would also be good to get agreement on the 
> implementation strategy.
> >
> >
> Absolutely - and I think we need to start on this ASAP...  
> There are certainly other people coding away furiously on 
> AMQP 1-0 implementations... and I think we at Apache should 
> be striving to be amongst the first (if not the first) to implement!
> 
> 
> > Personally what I believe we would want to do is provide 
> native Java, 
> > Python and C++ 1.0 transports.
> >
> > Additionally I would like to see that we can use either the 
> native or 
> > C++ transports in the Java and Python clients, and then I 
> assume that 
> > Ruby, PHP, C#, etc will rely on the C++ transport.
> >
> > This brings the advantage of being able to run pure Java, but also 
> > allows for Java to use the RDMA transports via C++.
> >
> >
> I'm fairly agnostic about a pure Python implementation... I 
> definitely see the need for a pure Java implementation, and a 
> widely portable C++ or C version (something that just works 
> across Windows, Mac, and all UNIX-like operating systems and 
> architectures).  Things like RDMA are likely to be less 
> portable, so wouldn't be in core, but should definitely be 
> available and pluggable into Java / .net / Scripting languages, etc.
> 
> 
> > This brings me to also believe that we need to introduce the new 
> > messaging API into the Java client, and then update the client to 
> > layer JMS over that. The new messaging
> > API would then be the layer at which we can swap the Java & 
> C++ impls out
> > under
> > the new messaging API.
> >
> >
> So - I think there are two different levels of abstraction we 
> need to draw here, there's the lower level transport, and the 
> higher level messaging API.  For something that will be of 
> use in brokers as well as clients, my feeling is that the 
> Messaging API is too high level and aimed more at client side 
> programming (although someone with more experience please 
> feel free to step in and correct me).
> 
> I think what we're saying is that first we build the 
> transport, and on top of that we build a a library that gives 
> the messaging API, and on top of that we build JMS, possible 
> WCF, and whatever other language-specific bindings make 
> sense.  The important thing is that these are independently 
> versioned, releasable components (even though we will 
> probably choose to release all at the same time in general).  
> This means that we're building infrastructure which is 
> generally useful for anybody who is interested in using or 
> *building* AMQP which should be an important part of our goal 
> as an Apache project aiming at widespread AMQP adoption.
> 
> I would see us building:
> 
> transport -  as a standalone library (with RDMA, SCTP,.. 
> whatever as possibly non-portable implementations of part of 
> that layer). messaging - a client api with sub-components for 
> SWIGed APIs in other language, as well as a native Java (and 
> possibly Python if desired) implementations jms - a JMS 
> facade that sits on top of a messaging implementation at the 
> same level we might have a wcf wrapper.
> 
> All of these should really be about speaking pure AMQP1-0 and 
> as such should be usable by *any* AMQP 1-0 broker (that is, 
> of course, the point of AMQP). Where we have special Qpid 
> goodness in our brokers they should be accessible through the 
> defined extension points in AMQP.
> 
> Brokers are a little more problematic as really we should be 
> looking to bring their functionality and (for want of a 
> better phrase) "user interface" more into line.  Again, I 
> think we want to be looking at reusing the transport 
> infrastructure (and again, the ability to use the C/C++ from 
> the Java Broker would be ideal).  We should also be looking 
> at separating out system tests into something that can be run 
> against either broker (or indeed any conforming AMQP 1-0 broker).
> 
> -- Rob
> 


---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Qpid and AMQP 1-0: Plans?

[Carl Trieloff <cc...@redhat.com>]

On 03/25/2011 10:49 AM, Robert Godfrey wrote:
> On 25 March 2011 14:57, Carl Trieloff<cc...@redhat.com>  wrote:
>
>>
>> Don't know if it is symantics but may not 100 agree, let's try pull that
>> apart.
>>
>>
>>
>>   I think what we're saying is that first we build the transport, and on top
>>> of that we build a a library that gives the messaging API, and on top of
>>> that we build JMS, possible WCF, and whatever other language-specific
>>> bindings make sense.  The important thing is that these are independently
>>> versioned, releasable components (even though we will probably choose to
>>> release all at the same time in general).
>>>
>>
>> I believe we have project goals&  AMQP goal. the project goals are not
>> necessarily
>> the exact same goals as AMQP.
>>
>> For eample, for Qpid users we want solid inter-op and upgrade as we
>> introduce AMQP
>> 1.0 and clean 0-10 1.0 inter-op for the C++ broker. For the Java broker
>> there may also
>> be a desire to also have 0-9 upgrade path. that is for those that know
>> better to comment
>> on.
>>
>> I would argue that this is the highest goal.
>>
>>
> I agree that the goals of Qpid and AMQP are not the same (and would actually
> add that there are a third set of goals - the commercial ones of companies
> building downstream products on top of Apache Qpid).  I would suggest though
> that the goal of Apache Qpid is not to be "Yet Another Message Broker", but
> it pretty squarely aimed at being a/the *AMQP* messaging solution.  Parts of
> the project have previously decided to change from one version of the
> protocol to another without having any sort of backwards API compatibility
> (e.g. I believe this was the case when the C++ broker went from speaking
> version 0.9 to 0.10) however I agree that this shouldn;t be the approach we
> take here.
>
>
>> Secondly we obviously want to make is most attractive for new use to be
>> AMQP 1.0 and
>> that we provide clean API's which we have the the messaging API to be able
>> to consume
>> it in many forms.
>>
>> Then there is an additional point of do we want to the 1.0 transport
>> semantics to be generally
>> reusable, which is a fine objective, but we should make sure it is fit for
>> Qpid propose first.
>>
>>
>>
> I think this is a matter of emphasis, and possibly also of philosophy.  To
> me we should be architecting in a way that allows us to see each strata of
> of implementation as a potentially independent component or library.
> Whether we actually spin these things off or not is essentially irrelevant.
> What we should be thinking is that whatever transport layer we write should
> be the best possible transport layer that anyone could write (because
> obviously we're smarter than other people ;-) )... Therefore lots of other
> people should want to use it too :-).
>
> Part of this is possibly due to the difference in design of AMQP1-0 over
> previous version of AMQP - namely that 1.0 is essentially symmetric and thus
> is makes more sense to share machinery between client and broker.
> Previously it would have been much less architecturally convenient to share
> between clients and brokers.  With 1-0 it becomes much more obvious that
> this is a shared component that can be maintained independently of clients
> or brokers that use it.



great, I think we are on the same page. my main point was that success 
for Qpid
is to be a fantastic widely used project, and having a re-able transport 
can be
an important part of that. but having a re-usable transport does not 
mean per
definition success for Qpid.



>
>>     This means that we're building
>>> infrastructure which is generally useful for anybody who is interested in
>>> using or *building* AMQP which should be an important part of our goal as
>>> an
>>> Apache project aiming at widespread AMQP adoption.
>>>
>>
>> I would ague that we should make our client, broker, install etc easy to
>> consume for
>> widespread adoption.
>>
>>
>>
> For widespread adoption I think we should be aiming at clients that work
> *any* AMQP 1-0 compliant broker, and brokers that work with *any* AMQP 1-0
> compliant client.  Anything less than that and we are missing the point of
> AMQP.  Clearly we are also interesting in building great broker(s) and we'll
> want those to out perform and out feature any other AMQP 1-0 broker out
> there... but there is no need for those features to be tied into our
> clients... and indeed it'll help us if people can use clients on platforms
> we don't support (for whatever reasons) but still use one of our superior
> brokers.
>
>

agree.


>>   I would see us building:
>>> transport -  as a standalone library (with RDMA, SCTP,.. whatever as
>>> possibly non-portable implementations of part of that layer).
>>> messaging - a client api with sub-components for SWIGed APIs in other
>>> language, as well as a native Java (and possibly Python if desired)
>>> implementations
>>> jms - a JMS facade that sits on top of a messaging implementation
>>> at the same level we might have a wcf wrapper.
>>>
>>> All of these should really be about speaking pure AMQP1-0 and as such
>>> should
>>> be usable by *any* AMQP 1-0 broker (that is, of course, the point of
>>> AMQP).
>>> Where we have special Qpid goodness in our brokers they should be
>>> accessible
>>> through the defined extension points in AMQP.
>>>
>>
>> Above you noted that we should start the transport. I believe for Java we
>> should
>> can work it from both sides and introduce the messaging API into the client
>> and
>> re-factor. Then we can bind in the transport down from the API and the work
>> on JMS can get done in parallel.
>>
>> For example, if we did the work on the API, which should follow the
>> existing pattern
>> we could refactor the JMS onto it and test it on the C++ swig bound Java
>> version on
>> existing 0-10. Then we have a stable point to work the java impl, C++ impl
>> for 1.0 and
>> JMS refactor in parallel.
>>
>> It also means that as the new transports get integrated into the brokers we
>> can get
>> wider test coverage faster for AMQP 1.0
>>
>>
>>
>>   Brokers are a little more problematic as really we should be looking to
>>> bring their functionality and (for want of a better phrase) "user
>>> interface"
>>> more into line.  Again, I think we want to be looking at reusing the
>>> transport infrastructure (and again, the ability to use the C/C++ from the
>>> Java Broker would be ideal).  We should also be looking at separating out
>>> system tests into something that can be run against either broker (or
>>> indeed
>>> any conforming AMQP 1-0 broker).
>>>
>> On the brokers I we are going to need to accept legacy and 1.0 traffic.
>>
>>
>>
> Absolutely - as previously mentioned I already have a prototype Java Broker
> that builds on top of our current 0-8/0-9/0-9-1/0-10 speaking one, and adds
> 1-0 support and message translation.
>
> -- Rob
>


---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Qpid and AMQP 1-0: Plans?

[Rajith Attapattu <ra...@gmail.com>]

I think we had a good discussion on this thread and some ideas/plans were
floated/discussed.
I am wondering where we are at in terms of the 1.0 plans ?

Maybe it's a good time to restart the discussion to get some ideas about
timelines or share any work that has been done in this area.

Rajith

On Tue, Mar 29, 2011 at 5:31 AM, Marnie McCormack <
marnie.mccormack@googlemail.com> wrote:

> <snip>
>
>  >
> > > For eample, for Qpid users we want solid inter-op and upgrade as we
> > > introduce AMQP
> > > 1.0 and clean 0-10 1.0 inter-op for the C++ broker. For the Java broker
> > > there may also
> > > be a desire to also have 0-9 upgrade path. that is for those that know
> > > better to comment
> > > on.
> > >
> >
> Based on my knowledge, I don't think we need a 0-9 upgrade path
> specifically
> - so long as on the Java side we continue to support the JMS semantics in
> the same way (with particular reference to client exception handling on
> send()). Perhaps more of a general issue is the gaps in feature sets as we
> move forward, which we'd need to consider for 1-0 and identify which are
> obstacles we should address (e.g. operational logging, flow control
> mechanisms, firewalls, acls etc).
>
> 0-10 traffic does need to be supported in a 1-0 broker though, I think.
>
> Hth,
> Marnie
>

Re: Qpid and AMQP 1-0: Plans?

[Marnie McCormack <ma...@googlemail.com>]

<snip>

 >
> > For eample, for Qpid users we want solid inter-op and upgrade as we
> > introduce AMQP
> > 1.0 and clean 0-10 1.0 inter-op for the C++ broker. For the Java broker
> > there may also
> > be a desire to also have 0-9 upgrade path. that is for those that know
> > better to comment
> > on.
> >
>
Based on my knowledge, I don't think we need a 0-9 upgrade path specifically
- so long as on the Java side we continue to support the JMS semantics in
the same way (with particular reference to client exception handling on
send()). Perhaps more of a general issue is the gaps in feature sets as we
move forward, which we'd need to consider for 1-0 and identify which are
obstacles we should address (e.g. operational logging, flow control
mechanisms, firewalls, acls etc).

0-10 traffic does need to be supported in a 1-0 broker though, I think.

Hth,
Marnie

Re: Qpid and AMQP 1-0: Plans?

[Robert Godfrey <ro...@gmail.com>]

On 25 March 2011 14:57, Carl Trieloff <cc...@redhat.com> wrote:

>
>
> Don't know if it is symantics but may not 100 agree, let's try pull that
> apart.
>
>
>
>  I think what we're saying is that first we build the transport, and on top
>> of that we build a a library that gives the messaging API, and on top of
>> that we build JMS, possible WCF, and whatever other language-specific
>> bindings make sense.  The important thing is that these are independently
>> versioned, releasable components (even though we will probably choose to
>> release all at the same time in general).
>>
>
>
> I believe we have project goals & AMQP goal. the project goals are not
> necessarily
> the exact same goals as AMQP.
>
> For eample, for Qpid users we want solid inter-op and upgrade as we
> introduce AMQP
> 1.0 and clean 0-10 1.0 inter-op for the C++ broker. For the Java broker
> there may also
> be a desire to also have 0-9 upgrade path. that is for those that know
> better to comment
> on.
>
> I would argue that this is the highest goal.
>
>

I agree that the goals of Qpid and AMQP are not the same (and would actually
add that there are a third set of goals - the commercial ones of companies
building downstream products on top of Apache Qpid).  I would suggest though
that the goal of Apache Qpid is not to be "Yet Another Message Broker", but
it pretty squarely aimed at being a/the *AMQP* messaging solution.  Parts of
the project have previously decided to change from one version of the
protocol to another without having any sort of backwards API compatibility
(e.g. I believe this was the case when the C++ broker went from speaking
version 0.9 to 0.10) however I agree that this shouldn;t be the approach we
take here.


> Secondly we obviously want to make is most attractive for new use to be
> AMQP 1.0 and
> that we provide clean API's which we have the the messaging API to be able
> to consume
> it in many forms.
>
> Then there is an additional point of do we want to the 1.0 transport
> semantics to be generally
> reusable, which is a fine objective, but we should make sure it is fit for
> Qpid propose first.
>
>
>
I think this is a matter of emphasis, and possibly also of philosophy.  To
me we should be architecting in a way that allows us to see each strata of
of implementation as a potentially independent component or library.
Whether we actually spin these things off or not is essentially irrelevant.
What we should be thinking is that whatever transport layer we write should
be the best possible transport layer that anyone could write (because
obviously we're smarter than other people ;-) )... Therefore lots of other
people should want to use it too :-).

Part of this is possibly due to the difference in design of AMQP1-0 over
previous version of AMQP - namely that 1.0 is essentially symmetric and thus
is makes more sense to share machinery between client and broker.
Previously it would have been much less architecturally convenient to share
between clients and brokers.  With 1-0 it becomes much more obvious that
this is a shared component that can be maintained independently of clients
or brokers that use it.


>
>    This means that we're building
>> infrastructure which is generally useful for anybody who is interested in
>> using or *building* AMQP which should be an important part of our goal as
>> an
>> Apache project aiming at widespread AMQP adoption.
>>
>
>
> I would ague that we should make our client, broker, install etc easy to
> consume for
> widespread adoption.
>
>
>
For widespread adoption I think we should be aiming at clients that work
*any* AMQP 1-0 compliant broker, and brokers that work with *any* AMQP 1-0
compliant client.  Anything less than that and we are missing the point of
AMQP.  Clearly we are also interesting in building great broker(s) and we'll
want those to out perform and out feature any other AMQP 1-0 broker out
there... but there is no need for those features to be tied into our
clients... and indeed it'll help us if people can use clients on platforms
we don't support (for whatever reasons) but still use one of our superior
brokers.


>
>  I would see us building:
>>
>> transport -  as a standalone library (with RDMA, SCTP,.. whatever as
>> possibly non-portable implementations of part of that layer).
>> messaging - a client api with sub-components for SWIGed APIs in other
>> language, as well as a native Java (and possibly Python if desired)
>> implementations
>> jms - a JMS facade that sits on top of a messaging implementation
>> at the same level we might have a wcf wrapper.
>>
>> All of these should really be about speaking pure AMQP1-0 and as such
>> should
>> be usable by *any* AMQP 1-0 broker (that is, of course, the point of
>> AMQP).
>> Where we have special Qpid goodness in our brokers they should be
>> accessible
>> through the defined extension points in AMQP.
>>
>
>
> Above you noted that we should start the transport. I believe for Java we
> should
> can work it from both sides and introduce the messaging API into the client
> and
> re-factor. Then we can bind in the transport down from the API and the work
> on JMS can get done in parallel.
>
> For example, if we did the work on the API, which should follow the
> existing pattern
> we could refactor the JMS onto it and test it on the C++ swig bound Java
> version on
> existing 0-10. Then we have a stable point to work the java impl, C++ impl
> for 1.0 and
> JMS refactor in parallel.
>
> It also means that as the new transports get integrated into the brokers we
> can get
> wider test coverage faster for AMQP 1.0
>
>
>
>  Brokers are a little more problematic as really we should be looking to
>> bring their functionality and (for want of a better phrase) "user
>> interface"
>> more into line.  Again, I think we want to be looking at reusing the
>> transport infrastructure (and again, the ability to use the C/C++ from the
>> Java Broker would be ideal).  We should also be looking at separating out
>> system tests into something that can be run against either broker (or
>> indeed
>> any conforming AMQP 1-0 broker).
>>
>
> On the brokers I we are going to need to accept legacy and 1.0 traffic.
>
>
>
Absolutely - as previously mentioned I already have a prototype Java Broker
that builds on top of our current 0-8/0-9/0-9-1/0-10 speaking one, and adds
1-0 support and message translation.

-- Rob

Re: Qpid and AMQP 1-0: Plans?

[Carl Trieloff <cc...@redhat.com>]

Don't know if it is symantics but may not 100 agree, let's try pull that 
apart.


> I think what we're saying is that first we build the transport, and on top
> of that we build a a library that gives the messaging API, and on top of
> that we build JMS, possible WCF, and whatever other language-specific
> bindings make sense.  The important thing is that these are independently
> versioned, releasable components (even though we will probably choose to
> release all at the same time in general).


I believe we have project goals & AMQP goal. the project goals are not 
necessarily
the exact same goals as AMQP.

For eample, for Qpid users we want solid inter-op and upgrade as we 
introduce AMQP
1.0 and clean 0-10 1.0 inter-op for the C++ broker. For the Java broker 
there may also
be a desire to also have 0-9 upgrade path. that is for those that know 
better to comment
on.

I would argue that this is the highest goal.

Secondly we obviously want to make is most attractive for new use to be 
AMQP 1.0 and
that we provide clean API's which we have the the messaging API to be 
able to consume
it in many forms.

Then there is an additional point of do we want to the 1.0 transport 
semantics to be generally
reusable, which is a fine objective, but we should make sure it is fit 
for Qpid propose first.


>    This means that we're building
> infrastructure which is generally useful for anybody who is interested in
> using or *building* AMQP which should be an important part of our goal as an
> Apache project aiming at widespread AMQP adoption.


I would ague that we should make our client, broker, install etc easy to 
consume for
widespread adoption.


> I would see us building:
>
> transport -  as a standalone library (with RDMA, SCTP,.. whatever as
> possibly non-portable implementations of part of that layer).
> messaging - a client api with sub-components for SWIGed APIs in other
> language, as well as a native Java (and possibly Python if desired)
> implementations
> jms - a JMS facade that sits on top of a messaging implementation
> at the same level we might have a wcf wrapper.
>
> All of these should really be about speaking pure AMQP1-0 and as such should
> be usable by *any* AMQP 1-0 broker (that is, of course, the point of AMQP).
> Where we have special Qpid goodness in our brokers they should be accessible
> through the defined extension points in AMQP.


Above you noted that we should start the transport. I believe for Java 
we should
can work it from both sides and introduce the messaging API into the 
client and
re-factor. Then we can bind in the transport down from the API and the work
on JMS can get done in parallel.

For example, if we did the work on the API, which should follow the 
existing pattern
we could refactor the JMS onto it and test it on the C++ swig bound Java 
version on
existing 0-10. Then we have a stable point to work the java impl, C++ 
impl for 1.0 and
JMS refactor in parallel.

It also means that as the new transports get integrated into the brokers 
we can get
wider test coverage faster for AMQP 1.0


> Brokers are a little more problematic as really we should be looking to
> bring their functionality and (for want of a better phrase) "user interface"
> more into line.  Again, I think we want to be looking at reusing the
> transport infrastructure (and again, the ability to use the C/C++ from the
> Java Broker would be ideal).  We should also be looking at separating out
> system tests into something that can be run against either broker (or indeed
> any conforming AMQP 1-0 broker).

On the brokers I we are going to need to accept legacy and 1.0 traffic.





---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Qpid and AMQP 1-0: Plans?

[Robert Godfrey <ro...@gmail.com>]

On 25 March 2011 13:44, Carl Trieloff <cc...@redhat.com> wrote:

>
>
> It would also be good to get agreement on the implementation strategy.
>
>
Absolutely - and I think we need to start on this ASAP...  There are
certainly other people coding away furiously on AMQP 1-0 implementations...
and I think we at Apache should be striving to be amongst the first (if not
the first) to implement!


> Personally what I believe we would want to do is provide native
> Java, Python and C++ 1.0 transports.
>
> Additionally I would like to see that we can use either the native or C++
> transports in the Java and Python clients, and then I assume that Ruby,
> PHP, C#, etc
> will rely on the C++ transport.
>
> This brings the advantage of being able to run pure Java, but also allows
> for Java
> to use the RDMA transports via C++.
>
>
I'm fairly agnostic about a pure Python implementation... I definitely see
the need for a pure Java implementation, and a widely portable C++ or C
version (something that just works across Windows, Mac, and all UNIX-like
operating systems and architectures).  Things like RDMA are likely to be
less portable, so wouldn't be in core, but should definitely be available
and pluggable into Java / .net / Scripting languages, etc.


> This brings me to also believe that we need to introduce the new messaging
> API into
> the Java client, and then update the client to layer JMS over that. The new
> messaging
> API would then be the layer at which we can swap the Java & C++ impls out
> under
> the new messaging API.
>
>
So - I think there are two different levels of abstraction we need to draw
here, there's the lower level transport, and the higher level messaging
API.  For something that will be of use in brokers as well as clients, my
feeling is that the Messaging API is too high level and aimed more at client
side programming (although someone with more experience please feel free to
step in and correct me).

I think what we're saying is that first we build the transport, and on top
of that we build a a library that gives the messaging API, and on top of
that we build JMS, possible WCF, and whatever other language-specific
bindings make sense.  The important thing is that these are independently
versioned, releasable components (even though we will probably choose to
release all at the same time in general).  This means that we're building
infrastructure which is generally useful for anybody who is interested in
using or *building* AMQP which should be an important part of our goal as an
Apache project aiming at widespread AMQP adoption.

I would see us building:

transport -  as a standalone library (with RDMA, SCTP,.. whatever as
possibly non-portable implementations of part of that layer).
messaging - a client api with sub-components for SWIGed APIs in other
language, as well as a native Java (and possibly Python if desired)
implementations
jms - a JMS facade that sits on top of a messaging implementation
at the same level we might have a wcf wrapper.

All of these should really be about speaking pure AMQP1-0 and as such should
be usable by *any* AMQP 1-0 broker (that is, of course, the point of AMQP).
Where we have special Qpid goodness in our brokers they should be accessible
through the defined extension points in AMQP.

Brokers are a little more problematic as really we should be looking to
bring their functionality and (for want of a better phrase) "user interface"
more into line.  Again, I think we want to be looking at reusing the
transport infrastructure (and again, the ability to use the C/C++ from the
Java Broker would be ideal).  We should also be looking at separating out
system tests into something that can be run against either broker (or indeed
any conforming AMQP 1-0 broker).

-- Rob

Re: Qpid and AMQP 1-0: Plans?

[Carl Trieloff <cc...@redhat.com>]

It would also be good to get agreement on the implementation strategy.

Personally what I believe we would want to do is provide native
Java, Python and C++ 1.0 transports.

Additionally I would like to see that we can use either the native or C++
transports in the Java and Python clients, and then I assume that Ruby, 
PHP, C#, etc
will rely on the C++ transport.

This brings the advantage of being able to run pure Java, but also 
allows for Java
to use the RDMA transports via C++.

This brings me to also believe that we need to introduce the new 
messaging API into
the Java client, and then update the client to layer JMS over that. The 
new messaging
API would then be the layer at which we can swap the Java & C++ impls 
out under
the new messaging API.

Carl.



On 03/25/2011 08:25 AM, Robert Godfrey wrote:
> On 25 March 2011 09:25, Andrew Kennedy<an...@gmail.com>wrote:
>
>> On 24 Mar 2011, at 16:45, Rafael Schloming wrote:
>>
>>> On 03/24/2011 11:55 AM, Andrew Stitcher wrote:
>>>
>>>> On Thu, 2011-03-24 at 16:07 +0100, Robert Godfrey wrote:
>>>>
>>>>> ...
>>>>> Branch is probably the wrong name here... but I'm thinking that this
>>>>> doesn't
>>>>> really fit under the same umbrella as the rest of the components which
>>>>> have
>>>>> a single release cycle currently.  What we possibly want is something
>>>>> that
>>>>> is in parallel to the existing codebase...
>>>>> repos/asf/qpid/trunk/transport in
>>>>> parallel to repos/asf/qpid/trunk/qpid would be one way of doing it...
>>>>> Although looking at other Apache projects it seems like the common way
>>>>> of
>>>>> doing this is
>>>>>
>>>>> .../asf/<project>/<component>/trunk or even
>>>>> .../asf/<project>/<component>/<sub-component>/trunk (see ant, commons,
>>>>> httpd, etc.)
>>>>>
>>>>> Though this would obviously require a little more re-organisation for us
>>>>> all...
>>>>>
>>>> Actually not necessarily, we already have a superfluous "qpid" directory
>>>> at the top level of trunk, so we could introduce "amqp-transport" at
>>>> that level if we wanted to.
>>>>
>>>> ie .../asf/qpid/trunk/amqp-transport/...
>>>>
>>>> I'm not necessarily advocating this, but it would avoid a whole lot of
>>>> top level moves that might heavily confuse the git mirroring process.
>>>>
>>> I'd be fine with this option as well. I think the key thing for me is to
>>> maintain the packaging and interface discipline of keeping the transport as
>>> a standalone entity.
>>>
>>
>> We could insert a sub-project at the top level while still maintaining the
>> existing structure? For example:
>>
>> .../asf/qpid/trunk/qpid/spec
>> .../asf/qpid/trunk/qpid/cpp
>> .../asf/qpid/trunk/qpid/java
>> .../asf/qpid/trunk/qpid/...
>>
>> .../asf/qpid/qpid-amqp/trunk/common
>> .../asf/qpid/qpid-amqp/trunk/cpp
>> .../asf/qpid/qpid-amqp/trunk/java
>> .../asf/qpid/qpid-amqp/trunk/...
>>
>> This is a fairly common idiom. ActiveMQ do this, for instance :
>>
>> http://svn.apache.org/viewvc/activemq/trunk/
>> http://svn.apache.org/viewvc/activemq/activemq-protobuf/trunk/
>>
> Andrew.
>
> Yeah - I think this is my preferred solution.
>
> To my mind there's probably scope for things that are currently under
> "trunk" to migrate to fully blown sub-projects over time, but I think the
> immediate next step would be to introduce the
>
> /asf/qpid/transport/[trunk|branches|...]
>
> structure and leave the rest of the project alone for now (especially since
> there should be no cross dependencies at this stage.
>
> -- Rob
>
>
>> --
>> -- andrew d kennedy ? do not fold, bend, spindle, or mutilate ;
>> -- http://grkvlt.blogspot.com/ ? edinburgh : +44 7582 293 255 ;
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project:      http://qpid.apache.org
>> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>>
>>


---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Qpid and AMQP 1-0: Plans?

[Robert Godfrey <ro...@gmail.com>]

On 25 March 2011 15:04, <jr...@redhat.com> wrote:

> On Fri, 25 Mar 2011, Robert Godfrey wrote:
>
>  We could insert a sub-project at the top level while still maintaining the
>>> existing structure? For example:
>>>
>>> .../asf/qpid/trunk/qpid/spec
>>> .../asf/qpid/trunk/qpid/cpp
>>> .../asf/qpid/trunk/qpid/java
>>> .../asf/qpid/trunk/qpid/...
>>>
>>> .../asf/qpid/qpid-amqp/trunk/common
>>> .../asf/qpid/qpid-amqp/trunk/cpp
>>> .../asf/qpid/qpid-amqp/trunk/java
>>> .../asf/qpid/qpid-amqp/trunk/...
>>>
>>> This is a fairly common idiom. ActiveMQ do this, for instance :
>>>
>>> http://svn.apache.org/viewvc/activemq/trunk/
>>> http://svn.apache.org/viewvc/activemq/activemq-protobuf/trunk/
>>>
>>>
>> Andrew.
>>
>>
>> Yeah - I think this is my preferred solution.
>>
>> To my mind there's probably scope for things that are currently under
>> "trunk" to migrate to fully blown sub-projects over time, but I think the
>> immediate next step would be to introduce the
>>
>> /asf/qpid/transport/[trunk|branches|...]
>>
>
> I don't object to subprojects (indeed, it's attractive in many ways), but I
> want us to have the bigger conversation about direction, first.
>
> The directories under qpid/trunk/qpid are already independent subprojects:
>
>  - They don't share code
>
>  - They don't share a build system or a test entry point
>
>  - They interface among themselves only optionally, just like independent
>    projects do
>
>  - They are largely worked on by independent groups of people
>
>
Actually, they are not really independent (which makes the situation
untenable)...  the directories have weird dependencies (possibly even
circular - though I haven't checked recently)... and they don't make
terribly much sense as independently releasable units.  They are generally
just broken down by language (which could be seen as a proxy for the group
of developers who work on them).


> What they share is apache infrastructure (jira category, mailing lists) and
> a release schedule.
>
> I don't like the pretending.  I want qpid to start to move in the direction
> of real integration (this has some very tangible benefits), or explicitly
> decide that these are independent (in code, apache infrastructure, and
> release cycle).
>
>
I agree we need to be clear in stating our goals.  In terms of integration
my focus would be on the user experience... that, to me, is what makes a
coherent project... Our internal development structure can follow our
external goals...

To me a coherent project would mean that

1) The clients have a common API
2) The brokers have similar functionality, configuration, etc ... and we
only need one set of docs to describe them

We're getting there (slowly) on 1) but 2) isn't all that great right now.
Moreover we're also including a whole lot of other stuff in our builds that
essentially should be seen (IMHO) as independent.



> Naturally, independent projects can have required dependencies, and under
> that plan we can increase our code sharing (relative to what we have now) as
> the transport intends to do.
>
> The downside of independent projects is, in a word, integration cost.
> Without a common release, source tree, build system, and test harness, the
> difficulty of maintaining a stable, coherent distribution is greater.
>
>
I actually disagree here as I think that the discipline of properly defining
the component entities that go into making a release will be a better way of
ensuring code sharing and test sharing.  At the moment we are siloed by
language... but there should be no reason why broker system tests (for
instance) are dependent on language.


> Indeed, that's the problem we face right now.  For instance, it's not easy
> *at all* for a qpid developer to test a change to one qpid component across
> the others.
>
>
So, to turn the question around... why should (s)he have to?  if you are
changing the transport you pass your own unit/system tests for that
component.  If one of the "users" of the transport then finds a bug, it
should be reported through JIRA - just as if an external project finds a
bug.  The problems come about when we have unnecessary coupling between
entities that should be distinct.  The Java codebase is terrible at this in
as much as we run tests that simultaneously test both the client and the
broker rather than having separate tests for each.

How do we want it to work?  How should we decide?
>
>
In my mind there are two aspects

Firstly we should all have a common idea about what the goal of the project
is, and should all be aiming towards that.
Secondly we need define the best structure that allows for each developer to
work on the components they wish to work on *without it unduly affecting
work elsewhere*.

Releasing all of Qpid in a single build/release should be easy if each
component is a self-contained and well tested unit.  On the other hand
(other than at major releases where there have been changes to the
interfaces/APIs) there's really no reason to have to release everything at
the same time.


-- Rob


> Justin
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>

Re: Qpid and AMQP 1-0: Plans?

[Rajith Attapattu <ra...@gmail.com>]

I think there at least 4 themes/sub-topics that have emerged out of this
discussion.

1. Common transport strategy
2. High level project goals (ex having a cohesive set of artefacts, reusing
components and minimising duplication).
3. Client/Broker Implementation strategies
4. Project source/component dir structure.

Perhaps it would be best if we spin off separate threads for each of these
sub-topics.
Then we could probably focus a bit more on each of those sub topics.

If people are happy with that I can volunteer to do start a thread on each
of those topics.

Rajith

On Fri, Mar 25, 2011 at 10:04 AM, <jr...@redhat.com> wrote:

> On Fri, 25 Mar 2011, Robert Godfrey wrote:
>
>  We could insert a sub-project at the top level while still maintaining the
>>> existing structure? For example:
>>>
>>> .../asf/qpid/trunk/qpid/spec
>>> .../asf/qpid/trunk/qpid/cpp
>>> .../asf/qpid/trunk/qpid/java
>>> .../asf/qpid/trunk/qpid/...
>>>
>>> .../asf/qpid/qpid-amqp/trunk/common
>>> .../asf/qpid/qpid-amqp/trunk/cpp
>>> .../asf/qpid/qpid-amqp/trunk/java
>>> .../asf/qpid/qpid-amqp/trunk/...
>>>
>>> This is a fairly common idiom. ActiveMQ do this, for instance :
>>>
>>> http://svn.apache.org/viewvc/activemq/trunk/
>>> http://svn.apache.org/viewvc/activemq/activemq-protobuf/trunk/
>>>
>>
>> Andrew.
>>
>>
>> Yeah - I think this is my preferred solution.
>>
>> To my mind there's probably scope for things that are currently under
>> "trunk" to migrate to fully blown sub-projects over time, but I think the
>> immediate next step would be to introduce the
>>
>> /asf/qpid/transport/[trunk|branches|...]
>>
>
> I don't object to subprojects (indeed, it's attractive in many ways), but I
> want us to have the bigger conversation about direction, first.
>
> The directories under qpid/trunk/qpid are already independent subprojects:
>
>  - They don't share code
>
>  - They don't share a build system or a test entry point
>
>  - They interface among themselves only optionally, just like independent
>    projects do
>
>  - They are largely worked on by independent groups of people
>
> What they share is apache infrastructure (jira category, mailing lists) and
> a release schedule.
>
> I don't like the pretending.  I want qpid to start to move in the direction
> of real integration (this has some very tangible benefits), or explicitly
> decide that these are independent (in code, apache infrastructure, and
> release cycle).
>
> Naturally, independent projects can have required dependencies, and under
> that plan we can increase our code sharing (relative to what we have now) as
> the transport intends to do.
>
> The downside of independent projects is, in a word, integration cost.
> Without a common release, source tree, build system, and test harness, the
> difficulty of maintaining a stable, coherent distribution is greater.
>
> Indeed, that's the problem we face right now.  For instance, it's not easy
> *at all* for a qpid developer to test a change to one qpid component across
> the others.
>
> How do we want it to work?  How should we decide?
>
> Justin
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>

Re: Qpid and AMQP 1-0: Plans?

[jr...@redhat.com]

On Fri, 25 Mar 2011, Robert Godfrey wrote:

>> We could insert a sub-project at the top level while still maintaining the
>> existing structure? For example:
>>
>> .../asf/qpid/trunk/qpid/spec
>> .../asf/qpid/trunk/qpid/cpp
>> .../asf/qpid/trunk/qpid/java
>> .../asf/qpid/trunk/qpid/...
>>
>> .../asf/qpid/qpid-amqp/trunk/common
>> .../asf/qpid/qpid-amqp/trunk/cpp
>> .../asf/qpid/qpid-amqp/trunk/java
>> .../asf/qpid/qpid-amqp/trunk/...
>>
>> This is a fairly common idiom. ActiveMQ do this, for instance :
>>
>> http://svn.apache.org/viewvc/activemq/trunk/
>> http://svn.apache.org/viewvc/activemq/activemq-protobuf/trunk/
>>
>
> Andrew.
>
>
> Yeah - I think this is my preferred solution.
>
> To my mind there's probably scope for things that are currently under
> "trunk" to migrate to fully blown sub-projects over time, but I think the
> immediate next step would be to introduce the
>
> /asf/qpid/transport/[trunk|branches|...]

I don't object to subprojects (indeed, it's attractive in many ways), but 
I want us to have the bigger conversation about direction, first.

The directories under qpid/trunk/qpid are already independent subprojects:

   - They don't share code

   - They don't share a build system or a test entry point

   - They interface among themselves only optionally, just like independent
     projects do

   - They are largely worked on by independent groups of people

What they share is apache infrastructure (jira category, mailing lists) 
and a release schedule.

I don't like the pretending.  I want qpid to start to move in the 
direction of real integration (this has some very tangible benefits), 
or explicitly decide that these are independent (in code, apache 
infrastructure, and release cycle).

Naturally, independent projects can have required dependencies, and under 
that plan we can increase our code sharing (relative to what we have now) 
as the transport intends to do.

The downside of independent projects is, in a word, integration cost. 
Without a common release, source tree, build system, and test harness, the 
difficulty of maintaining a stable, coherent distribution is greater.

Indeed, that's the problem we face right now.  For instance, it's not easy 
*at all* for a qpid developer to test a change to one qpid component 
across the others.

How do we want it to work?  How should we decide?

Justin

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Qpid and AMQP 1-0: Plans?

[Robert Godfrey <ro...@gmail.com>]

On 25 March 2011 09:25, Andrew Kennedy <an...@gmail.com>wrote:

> On 24 Mar 2011, at 16:45, Rafael Schloming wrote:
>
>> On 03/24/2011 11:55 AM, Andrew Stitcher wrote:
>>
>>> On Thu, 2011-03-24 at 16:07 +0100, Robert Godfrey wrote:
>>>
>>>> ...
>>>> Branch is probably the wrong name here... but I'm thinking that this
>>>> doesn't
>>>> really fit under the same umbrella as the rest of the components which
>>>> have
>>>> a single release cycle currently.  What we possibly want is something
>>>> that
>>>> is in parallel to the existing codebase...
>>>> repos/asf/qpid/trunk/transport in
>>>> parallel to repos/asf/qpid/trunk/qpid would be one way of doing it...
>>>> Although looking at other Apache projects it seems like the common way
>>>> of
>>>> doing this is
>>>>
>>>> .../asf/<project>/<component>/trunk or even
>>>> .../asf/<project>/<component>/<sub-component>/trunk (see ant, commons,
>>>> httpd, etc.)
>>>>
>>>> Though this would obviously require a little more re-organisation for us
>>>> all...
>>>>
>>>
>>> Actually not necessarily, we already have a superfluous "qpid" directory
>>> at the top level of trunk, so we could introduce "amqp-transport" at
>>> that level if we wanted to.
>>>
>>> ie .../asf/qpid/trunk/amqp-transport/...
>>>
>>> I'm not necessarily advocating this, but it would avoid a whole lot of
>>> top level moves that might heavily confuse the git mirroring process.
>>>
>>
>> I'd be fine with this option as well. I think the key thing for me is to
>> maintain the packaging and interface discipline of keeping the transport as
>> a standalone entity.
>>
>
>
> We could insert a sub-project at the top level while still maintaining the
> existing structure? For example:
>
> .../asf/qpid/trunk/qpid/spec
> .../asf/qpid/trunk/qpid/cpp
> .../asf/qpid/trunk/qpid/java
> .../asf/qpid/trunk/qpid/...
>
> .../asf/qpid/qpid-amqp/trunk/common
> .../asf/qpid/qpid-amqp/trunk/cpp
> .../asf/qpid/qpid-amqp/trunk/java
> .../asf/qpid/qpid-amqp/trunk/...
>
> This is a fairly common idiom. ActiveMQ do this, for instance :
>
> http://svn.apache.org/viewvc/activemq/trunk/
> http://svn.apache.org/viewvc/activemq/activemq-protobuf/trunk/
>

Andrew.
>


Yeah - I think this is my preferred solution.

To my mind there's probably scope for things that are currently under
"trunk" to migrate to fully blown sub-projects over time, but I think the
immediate next step would be to introduce the

/asf/qpid/transport/[trunk|branches|...]

structure and leave the rest of the project alone for now (especially since
there should be no cross dependencies at this stage.

-- Rob


> --
> -- andrew d kennedy ? do not fold, bend, spindle, or mutilate ;
> -- http://grkvlt.blogspot.com/ ? edinburgh : +44 7582 293 255 ;
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>

Re: Qpid and AMQP 1-0: Plans?

[Andrew Kennedy <an...@gmail.com>]

On 24 Mar 2011, at 16:45, Rafael Schloming wrote:
> On 03/24/2011 11:55 AM, Andrew Stitcher wrote:
>> On Thu, 2011-03-24 at 16:07 +0100, Robert Godfrey wrote:
>>> ...
>>> Branch is probably the wrong name here... but I'm thinking that  
>>> this doesn't
>>> really fit under the same umbrella as the rest of the components  
>>> which have
>>> a single release cycle currently.  What we possibly want is  
>>> something that
>>> is in parallel to the existing codebase... repos/asf/qpid/trunk/ 
>>> transport in
>>> parallel to repos/asf/qpid/trunk/qpid would be one way of doing  
>>> it...
>>> Although looking at other Apache projects it seems like the  
>>> common way of
>>> doing this is
>>>
>>> .../asf/<project>/<component>/trunk or even
>>> .../asf/<project>/<component>/<sub-component>/trunk (see ant,  
>>> commons,
>>> httpd, etc.)
>>>
>>> Though this would obviously require a little more re-organisation  
>>> for us
>>> all...
>>
>> Actually not necessarily, we already have a superfluous "qpid"  
>> directory
>> at the top level of trunk, so we could introduce "amqp-transport" at
>> that level if we wanted to.
>>
>> ie .../asf/qpid/trunk/amqp-transport/...
>>
>> I'm not necessarily advocating this, but it would avoid a whole  
>> lot of
>> top level moves that might heavily confuse the git mirroring process.
>
> I'd be fine with this option as well. I think the key thing for me  
> is to maintain the packaging and interface discipline of keeping  
> the transport as a standalone entity.


We could insert a sub-project at the top level while still  
maintaining the existing structure? For example:

.../asf/qpid/trunk/qpid/spec
.../asf/qpid/trunk/qpid/cpp
.../asf/qpid/trunk/qpid/java
.../asf/qpid/trunk/qpid/...

.../asf/qpid/qpid-amqp/trunk/common
.../asf/qpid/qpid-amqp/trunk/cpp
.../asf/qpid/qpid-amqp/trunk/java
.../asf/qpid/qpid-amqp/trunk/...

This is a fairly common idiom. ActiveMQ do this, for instance:

http://svn.apache.org/viewvc/activemq/trunk/
http://svn.apache.org/viewvc/activemq/activemq-protobuf/trunk/

Andrew.
-- 
-- andrew d kennedy ? do not fold, bend, spindle, or mutilate ;
-- http://grkvlt.blogspot.com/ ? edinburgh : +44 7582 293 255 ;
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Qpid and AMQP 1-0: Plans?

[Rafael Schloming <ra...@redhat.com>]

On 03/24/2011 11:55 AM, Andrew Stitcher wrote:
> On Thu, 2011-03-24 at 16:07 +0100, Robert Godfrey wrote:
>> ...
>> Branch is probably the wrong name here... but I'm thinking that this doesn't
>> really fit under the same umbrella as the rest of the components which have
>> a single release cycle currently.  What we possibly want is something that
>> is in parallel to the existing codebase... repos/asf/qpid/trunk/transport in
>> parallel to repos/asf/qpid/trunk/qpid would be one way of doing it...
>> Although looking at other Apache projects it seems like the common way of
>> doing this is
>>
>> .../asf/<project>/<component>/trunk or even
>> .../asf/<project>/<component>/<sub-component>/trunk (see ant, commons,
>> httpd, etc.)
>>
>> Though this would obviously require a little more re-organisation for us
>> all...
>
> Actually not necessarily, we already have a superfluous "qpid" directory
> at the top level of trunk, so we could introduce "amqp-transport" at
> that level if we wanted to.
>
> ie .../asf/qpid/trunk/amqp-transport/...
>
> I'm not necessarily advocating this, but it would avoid a whole lot of
> top level moves that might heavily confuse the git mirroring process.

I'd be fine with this option as well. I think the key thing for me is to 
maintain the packaging and interface discipline of keeping the transport 
as a standalone entity.

--Rafael

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Qpid and AMQP 1-0: Plans?

[Andrew Stitcher <as...@redhat.com>]

On Thu, 2011-03-24 at 16:07 +0100, Robert Godfrey wrote:
> ...
> Branch is probably the wrong name here... but I'm thinking that this doesn't
> really fit under the same umbrella as the rest of the components which have
> a single release cycle currently.  What we possibly want is something that
> is in parallel to the existing codebase... repos/asf/qpid/trunk/transport in
> parallel to repos/asf/qpid/trunk/qpid would be one way of doing it...
> Although looking at other Apache projects it seems like the common way of
> doing this is
> 
> .../asf/<project>/<component>/trunk or even
> .../asf/<project>/<component>/<sub-component>/trunk (see ant, commons,
> httpd, etc.)
> 
> Though this would obviously require a little more re-organisation for us
> all...

Actually not necessarily, we already have a superfluous "qpid" directory
at the top level of trunk, so we could introduce "amqp-transport" at
that level if we wanted to.

ie .../asf/qpid/trunk/amqp-transport/...

I'm not necessarily advocating this, but it would avoid a whole lot of
top level moves that might heavily confuse the git mirroring process.

[BTW sorry to spoil the double act]

Andrew



---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Qpid and AMQP 1-0: Plans?

[Robert Godfrey <ro...@gmail.com>]

On 24 March 2011 15:56, Rafael Schloming <ra...@redhat.com> wrote:

> On 03/24/2011 10:43 AM, Robert Godfrey wrote:
>
>> So - obviously we need to hear a few more voices on this... But personally
>> I'm *really* keen to get the project moving towards AMQP 1-0 ASAP.  In my
>> spare time I've sort of hacked up an AMQP 1-0 Java client, and then
>> spliced
>> that onto the Qpid Java Broker...  It's not code I would really consider
>> putting into Qpid, but this (and the code you've got on github) may be
>> able
>> to help us bootstrap testing our 1-0 efforts.
>>
>> If we follow the idea of the common transport, I would guess that the best
>> way to progress this is probably to create a branch where we can start out
>> defining and implementing this transport as a standalone entity.  It
>> doesn;t
>> seem to make sense to me to have this in trunk and tie it into the release
>> cycles of the current Qpid work...
>>
>> Thoughts?
>>
>
> For developing the transport itself, I'm not sure it matters much whether
> we branch or not if we're treating it as an independent component. Obviously
> if we start integrating it into other pieces then a branch for those pieces
> would be called for. The release cycle question is certainly interesting. I
> agree it shouldn't really be tied to the trunk release in one sense, but I
> do think it would be good to have something ready for this year's ApacheCon,
> and given that it might make sense to have deliverable milestones that
> coincide with the trunk releases purely as a means of focusing progress.
>
>
>From a theoretical point of view I agree.

Logically we are starting from a clean slate and developing what amounts to
an independent library which we will then, hopefully, make into a dependency
of the rest of qpid...

Branch is probably the wrong name here... but I'm thinking that this doesn't
really fit under the same umbrella as the rest of the components which have
a single release cycle currently.  What we possibly want is something that
is in parallel to the existing codebase... repos/asf/qpid/trunk/transport in
parallel to repos/asf/qpid/trunk/qpid would be one way of doing it...
Although looking at other Apache projects it seems like the common way of
doing this is

.../asf/<project>/<component>/trunk or even
.../asf/<project>/<component>/<sub-component>/trunk (see ant, commons,
httpd, etc.)

Though this would obviously require a little more re-organisation for us
all...

-- Rob

Re: Qpid and AMQP 1-0: Plans?

[Rafael Schloming <ra...@redhat.com>]

On 03/24/2011 10:43 AM, Robert Godfrey wrote:
> So - obviously we need to hear a few more voices on this... But personally
> I'm *really* keen to get the project moving towards AMQP 1-0 ASAP.  In my
> spare time I've sort of hacked up an AMQP 1-0 Java client, and then spliced
> that onto the Qpid Java Broker...  It's not code I would really consider
> putting into Qpid, but this (and the code you've got on github) may be able
> to help us bootstrap testing our 1-0 efforts.
>
> If we follow the idea of the common transport, I would guess that the best
> way to progress this is probably to create a branch where we can start out
> defining and implementing this transport as a standalone entity.  It doesn;t
> seem to make sense to me to have this in trunk and tie it into the release
> cycles of the current Qpid work...
>
> Thoughts?

For developing the transport itself, I'm not sure it matters much 
whether we branch or not if we're treating it as an independent 
component. Obviously if we start integrating it into other pieces then a 
branch for those pieces would be called for. The release cycle question 
is certainly interesting. I agree it shouldn't really be tied to the 
trunk release in one sense, but I do think it would be good to have 
something ready for this year's ApacheCon, and given that it might make 
sense to have deliverable milestones that coincide with the trunk 
releases purely as a means of focusing progress.

--Rafael

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Qpid and AMQP 1-0: Plans?

[Robert Godfrey <ro...@gmail.com>]

On 24 March 2011 15:34, Rafael Schloming <ra...@redhat.com> wrote:

> On 03/24/2011 10:25 AM, Robert Godfrey wrote:
>
>> On 24 March 2011 15:11, Rafael Schloming<ra...@redhat.com>  wrote:
>>
>>  On 03/24/2011 09:57 AM, Robert Godfrey wrote:
>>>
>>>  On 24 March 2011 14:47, Rafael Schloming<ra...@redhat.com>   wrote:
>>>>
>>>>
>>>>
>>>  So - as above... this sounds like the beginning of a plan.  I really
>> need to
>> have a better idea of the sort of interfaces you are talking about between
>> the different components of your architecture... do you think it would be
>> useful to produce some sort of UML diagram showing how you thing the
>> interfaces between these would look?
>>
>
> Yes, I think that would be useful. In fact I've already taken a very brief
> look at umlgraph which is purported to be able to automatically produce UML
> diagrams from annotated java code. From what I understand it's basically
> just a doclet that processes the class hierarchy and produces a graphviz
> representation. This might be a good place to start, it would be easy to
> throw together a few skeletal java classes that match the interfaces
> currently in the python prototype.
>
>
>
So - obviously we need to hear a few more voices on this... But personally
I'm *really* keen to get the project moving towards AMQP 1-0 ASAP.  In my
spare time I've sort of hacked up an AMQP 1-0 Java client, and then spliced
that onto the Qpid Java Broker...  It's not code I would really consider
putting into Qpid, but this (and the code you've got on github) may be able
to help us bootstrap testing our 1-0 efforts.

If we follow the idea of the common transport, I would guess that the best
way to progress this is probably to create a branch where we can start out
defining and implementing this transport as a standalone entity.  It doesn;t
seem to make sense to me to have this in trunk and tie it into the release
cycles of the current Qpid work...

Thoughts?

-- Rob

Re: Qpid and AMQP 1-0: Plans?

[Andrew Kennedy <an...@gmail.com>]

On 24 Mar 2011, at 14:34, Rafael Schloming wrote:
> Yes, I think that would be useful. In fact I've already taken a  
> very brief look at umlgraph which is purported to be able to  
> automatically produce UML diagrams from annotated java code. From  
> what I understand it's basically just a doclet that processes the  
> class hierarchy and produces a graphviz representation. This might  
> be a good place to start, it would be easy to throw together a few  
> skeletal java classes that match the interfaces currently in the  
> python prototype.

I have also used PlantUML [1] to successfully document some of the  
0-10 Java classes, adding sequence diagrams to the existing Javadoc.  
UMLGraph [2] looks very similar. Either way, formal notation like  
this in a language agnostic format is the way forward for documentation.

[1] http://plantuml.sourceforge.net/
[2] http://www.umlgraph.org/

Andrew.
-- 
-- andrew d kennedy ? do not fold, bend, spindle, or mutilate ;
-- http://grkvlt.blogspot.com/ ? edinburgh : +44 7582 293 255 ;

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Qpid and AMQP 1-0: Plans?

[Rafael Schloming <ra...@redhat.com>]

On 03/24/2011 10:25 AM, Robert Godfrey wrote:
> On 24 March 2011 15:11, Rafael Schloming<ra...@redhat.com>  wrote:
>
>> On 03/24/2011 09:57 AM, Robert Godfrey wrote:
>>
>>> On 24 March 2011 14:47, Rafael Schloming<ra...@redhat.com>   wrote:
>>>
>>>
> *snip*
>
>
>>
>> I agree with your suggested goal. I would actually go slightly farther and
>> say that pragmatically to succeed we need a much less confusing story for
>> users than we have right now, so a further goal for the 1-0 work needs to be
>> to supplant the many obviously separate parts of qpid with a more coherent
>> suite of artifacts.
>>
>>
> I don't want to bite off more than we can chew - but I certainly agree that
> we need to make Qpid a much more coherent story than it is now.  However,
> possibly that's a slightly different, though equally important, objective to
> the one we seem to be discussing now.  If we can direct our energies so that
> the necessary energies we need to expend to get Qpid speaking AMQP 1-0, with
> the necessary number of client language bindings... and at the same time
> increase the commonality of API, functionality and configuration for other
> artefacts (brokers in particular) that would be a spectacular success.
>
> I does seem to me though that we should first set our sights firmly on
> achievable goals... and defining and implementing a "common transport layer"
> (even if to start off with it is not actually in use in the current
> artefacts) is a very achievable goal.
>
>
>> As for what I mean by a common transport layer, I think it’s useful to
>> consider two different dimensions of the problem: firstly, sharing a
>> transport implementation between different programming models within a
>> homogenous environment, e.g. a broker and a client, and secondly sharing an
>> implementation between heterogenous environments, i.e. different languages.
>>
>> I’ve attached a presentation that introduces some of my ideas at a high
>> level. I think some people have seen this already, but it's probably a good
>> time to share with a broader audience. I’ve also been doing some prototyping
>> work exploring an architecture that addresses both of these points in the
>> context of a 1-0 implementation. I have a more complete python version and a
>> less complete (but improving) C version, both currently hosted at github:
>>
>> https://github.com/rhs/amqp (python prototype)
>> https://github.com/rhs/amp (C prototype)
>>
>> Note that the github location is purely for convenience of sharing and
>> backup until I have something that will make sense to introduce to qpid.
>>
>> As for how to incorporate this into qpid, I think if we want to become a
>> ubiquitous protocol implementation, then the transport has to be an
>> independently available component that is easily embeddable within third
>> party apps that wish to integrate with messaging. As such it needs to have
>> as few dependencies as possible and a very narrow and well considered set of
>> interfaces. Given this, the natural way to develop this would be as an
>> independent sub project within qpid, so that from the start we are forced to
>> observe good packaging and dependency practices and consider our interfaces
>> carefully.
>>
>>
> So - as above... this sounds like the beginning of a plan.  I really need to
> have a better idea of the sort of interfaces you are talking about between
> the different components of your architecture... do you think it would be
> useful to produce some sort of UML diagram showing how you thing the
> interfaces between these would look?

Yes, I think that would be useful. In fact I've already taken a very 
brief look at umlgraph which is purported to be able to automatically 
produce UML diagrams from annotated java code. From what I understand 
it's basically just a doclet that processes the class hierarchy and 
produces a graphviz representation. This might be a good place to start, 
it would be easy to throw together a few skeletal java classes that 
match the interfaces currently in the python prototype.

--Rafael

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Qpid and AMQP 1-0: Plans?

[Robert Godfrey <ro...@gmail.com>]

On 24 March 2011 15:11, Rafael Schloming <ra...@redhat.com> wrote:

> On 03/24/2011 09:57 AM, Robert Godfrey wrote:
>
>> On 24 March 2011 14:47, Rafael Schloming<ra...@redhat.com>  wrote:
>>
>>
*snip*


>
> I agree with your suggested goal. I would actually go slightly farther and
> say that pragmatically to succeed we need a much less confusing story for
> users than we have right now, so a further goal for the 1-0 work needs to be
> to supplant the many obviously separate parts of qpid with a more coherent
> suite of artifacts.
>
>
I don't want to bite off more than we can chew - but I certainly agree that
we need to make Qpid a much more coherent story than it is now.  However,
possibly that's a slightly different, though equally important, objective to
the one we seem to be discussing now.  If we can direct our energies so that
the necessary energies we need to expend to get Qpid speaking AMQP 1-0, with
the necessary number of client language bindings... and at the same time
increase the commonality of API, functionality and configuration for other
artefacts (brokers in particular) that would be a spectacular success.

I does seem to me though that we should first set our sights firmly on
achievable goals... and defining and implementing a "common transport layer"
(even if to start off with it is not actually in use in the current
artefacts) is a very achievable goal.


> As for what I mean by a common transport layer, I think it’s useful to
> consider two different dimensions of the problem: firstly, sharing a
> transport implementation between different programming models within a
> homogenous environment, e.g. a broker and a client, and secondly sharing an
> implementation between heterogenous environments, i.e. different languages.
>
> I’ve attached a presentation that introduces some of my ideas at a high
> level. I think some people have seen this already, but it's probably a good
> time to share with a broader audience. I’ve also been doing some prototyping
> work exploring an architecture that addresses both of these points in the
> context of a 1-0 implementation. I have a more complete python version and a
> less complete (but improving) C version, both currently hosted at github:
>
> https://github.com/rhs/amqp (python prototype)
> https://github.com/rhs/amp (C prototype)
>
> Note that the github location is purely for convenience of sharing and
> backup until I have something that will make sense to introduce to qpid.
>
> As for how to incorporate this into qpid, I think if we want to become a
> ubiquitous protocol implementation, then the transport has to be an
> independently available component that is easily embeddable within third
> party apps that wish to integrate with messaging. As such it needs to have
> as few dependencies as possible and a very narrow and well considered set of
> interfaces. Given this, the natural way to develop this would be as an
> independent sub project within qpid, so that from the start we are forced to
> observe good packaging and dependency practices and consider our interfaces
> carefully.
>
>
So - as above... this sounds like the beginning of a plan.  I really need to
have a better idea of the sort of interfaces you are talking about between
the different components of your architecture... do you think it would be
useful to produce some sort of UML diagram showing how you thing the
interfaces between these would look?

-- Rob


> --Rafael
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>

Re: Qpid and AMQP 1-0: Plans?

[Robert Godfrey <ro...@gmail.com>]

I'm planning on checking in all the Java code I've been working on for AMQP
1-0 on a branch soon... possibly over the weekend... it's very rough, and I
would like to remodel it in terms of the way we've discussed... however the
broker changes allow the Qpid Java Broker to do a passable impression of an
AMQP 1-0 broker.  For a client I've been building something standalone as
the current Java client is ... how to put this politely ... not the easiest
place to start implementing from :-)

What I'll probably start doing soon is working on a new implementation that
follows the style we have discussed below and picking up the things I can
re-use from my (essentially) prototyping work.

Cheers,
Rob

On 12 August 2011 16:49, Gordon Sim <gs...@redhat.com> wrote:

> On 03/24/2011 02:11 PM, Rafael Schloming wrote:
>
>> I’ve attached a presentation that introduces some of my ideas at a high
>> level. I think some people have seen this already, but it's probably a
>> good time to share with a broader audience. I’ve also been doing some
>> prototyping work exploring an architecture that addresses both of these
>> points in the context of a 1-0 implementation. I have a more complete
>> python version and a less complete (but improving) C version, both
>> currently hosted at github:
>>
>> https://github.com/rhs/amqp (python prototype)
>> https://github.com/rhs/amp (C prototype)
>>
>> Note that the github location is purely for convenience of sharing and
>> backup until I have something that will make sense to introduce to qpid.
>>
>> As for how to incorporate this into qpid, I think if we want to become a
>> ubiquitous protocol implementation, then the transport has to be an
>> independently available component that is easily embeddable within third
>> party apps that wish to integrate with messaging. As such it needs to
>> have as few dependencies as possible and a very narrow and well
>> considered set of interfaces. Given this, the natural way to develop
>> this would be as an independent sub project within qpid, so that from
>> the start we are forced to observe good packaging and dependency
>> practices and consider our interfaces carefully.
>>
>
> To start getting some more insight into how this approach might look, I
> have been doing some hacking on Rafi's prototype C engine and on an
> implementation of the Qpid C++ messaging API over it as well as a first stab
> at integrating it with the C++ broker.
>
> What I've got so far is very rough and very incomplete. However it does
> allow the familiar drain/spout examples to be run using the messaging API
> against the C++ broker using the latest 1.0 protocol. I've only tested it
> against Rafi's python engine so far apart from that.
>
> This approach looks promising to me. I think it would be great to start
> getting some momentum on 1.0 within Qpid over the next couple of months.
> Whether we will be in a position to have support of any description in the
> 0.14 or whether we simply have some previews available in that time frame, I
> think we do want to demonstrate that we are working towards it.
>
> I plan to work with Rafi to get his engine onto a branch in Qpid's svn and
> then (again on a branch) to progress the integration of that into the C++
> components. I'd welcome any input from other interested parties and a branch
> seems the simplest way to facilitate that. In the mean time I've attached my
> latest patches for qpid svn and the engine itself.
>
> I'll stress again that these are very much early works in progress. The
> engine changes are as yet unreviewed by Rafi and I imagine will undergo some
> changes once that review takes place. The integration pieces on the client
> and broker are also very incomplete.
>
> I've also attached some notes on the most obvious missing pieces at
> present. There is a lot of work affected by the latest protocol. However we
> don't in my view need to tackle it all at once. Hopefully these notes will
> be useful in starting some planning discussions.
>
> I'll send another mail once I get this all on to a branch in svn. This is
> just an early notice of that intent.
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>

Re: Qpid and AMQP 1-0: Plans?

[Gordon Sim <gs...@redhat.com>]

On 03/24/2011 02:11 PM, Rafael Schloming wrote:
> I’ve attached a presentation that introduces some of my ideas at a high
> level. I think some people have seen this already, but it's probably a
> good time to share with a broader audience. I’ve also been doing some
> prototyping work exploring an architecture that addresses both of these
> points in the context of a 1-0 implementation. I have a more complete
> python version and a less complete (but improving) C version, both
> currently hosted at github:
>
> https://github.com/rhs/amqp (python prototype)
> https://github.com/rhs/amp (C prototype)
>
> Note that the github location is purely for convenience of sharing and
> backup until I have something that will make sense to introduce to qpid.
>
> As for how to incorporate this into qpid, I think if we want to become a
> ubiquitous protocol implementation, then the transport has to be an
> independently available component that is easily embeddable within third
> party apps that wish to integrate with messaging. As such it needs to
> have as few dependencies as possible and a very narrow and well
> considered set of interfaces. Given this, the natural way to develop
> this would be as an independent sub project within qpid, so that from
> the start we are forced to observe good packaging and dependency
> practices and consider our interfaces carefully.

To start getting some more insight into how this approach might look, I 
have been doing some hacking on Rafi's prototype C engine and on an 
implementation of the Qpid C++ messaging API over it as well as a first 
stab at integrating it with the C++ broker.

What I've got so far is very rough and very incomplete. However it does 
allow the familiar drain/spout examples to be run using the messaging 
API against the C++ broker using the latest 1.0 protocol. I've only 
tested it against Rafi's python engine so far apart from that.

This approach looks promising to me. I think it would be great to start 
getting some momentum on 1.0 within Qpid over the next couple of months. 
Whether we will be in a position to have support of any description in 
the 0.14 or whether we simply have some previews available in that time 
frame, I think we do want to demonstrate that we are working towards it.

I plan to work with Rafi to get his engine onto a branch in Qpid's svn 
and then (again on a branch) to progress the integration of that into 
the C++ components. I'd welcome any input from other interested parties 
and a branch seems the simplest way to facilitate that. In the mean time 
I've attached my latest patches for qpid svn and the engine itself.

I'll stress again that these are very much early works in progress. The 
engine changes are as yet unreviewed by Rafi and I imagine will undergo 
some changes once that review takes place. The integration pieces on the 
client and broker are also very incomplete.

I've also attached some notes on the most obvious missing pieces at 
present. There is a lot of work affected by the latest protocol. However 
we don't in my view need to tackle it all at once. Hopefully these notes 
will be useful in starting some planning discussions.

I'll send another mail once I get this all on to a branch in svn. This 
is just an early notice of that intent.

Re: Qpid and AMQP 1-0: Plans?

[Rafael Schloming <ra...@redhat.com>]

On 03/24/2011 09:57 AM, Robert Godfrey wrote:
> On 24 March 2011 14:47, Rafael Schloming<ra...@redhat.com>  wrote:
>
>> On 03/24/2011 09:42 AM, Robert Godfrey wrote:
>>
>> I think one of the most important factors to consider is how we build out
>> protocol support in a way that avoids some of the significant
>> maintainability issues we’ve had with our previous approach. It’s pretty
>> clear that one of our primary pain points has been with client support. The
>> practice of transcoding wholesale from one client language into another has
>> resulted in a significant number of write-only implementations that have
>> proven impractical to maintain.
>>
>> IMHO, one of the key elements of our 1-0 strategy going forward has to be
>> to share the minimum possible number of protocol implementations between as
>> many different client and broker environments as possible. This goal has in
>> fact been a key principle in the design of the 1-0 version of the protocol.
>> Its symmetry and layering is specifically designed to allow for the majority
>> of the protocol machinery to be encapsulated within a single transport layer
>> with a simple interface that can be embedded within a wide variety of
>> environments.
>>
>>
> I agree with this.  The idea of a common transport layer is one that has
> been raised before and which I think had broad support.  To me, the primary
> goal of the Apache Qpid Project must be to aim to be the de facto standard
> implementation of AMQP.  In order to achieve this goal we need to be able to
> provide client APIs across as many languages as possible (and across OS and
> machine architectures as well).  We can already see that our current
> approach to this is not sustainable - we’ve had suggestions that we should
> drop a number of client API artefacts from the 0.10 release as they are no
> longer able to be maintained.
>
> Having said that I think we need a little more clarity in defining what we
> mean by a “common transport layer”.  Firstly, at what level do you see the
> API here - is it simply an implementation of the AMQP 1-0 protocol, or are
> we aiming at a higher abstraction that will allow us to have implementations
> of the “transport layer” which use AMQP 0-10, 0-9-1 etc. as their protocol?
> Secondly, if we are looking to share the transport layer between “clients”
> and “brokers” it will need to be designed in such a way that the very
> different programming models (by which I really mean threading models) work.
>
> Lastly, and I guess most crucially, how do we see this work being
> incorporated into our current Qpid implementations?

I agree with your suggested goal. I would actually go slightly farther 
and say that pragmatically to succeed we need a much less confusing 
story for users than we have right now, so a further goal for the 1-0 
work needs to be to supplant the many obviously separate parts of qpid 
with a more coherent suite of artifacts.

As for what I mean by a common transport layer, I think it’s useful to 
consider two different dimensions of the problem: firstly, sharing a 
transport implementation between different programming models within a 
homogenous environment, e.g. a broker and a client, and secondly sharing 
an implementation between heterogenous environments, i.e. different 
languages.

I’ve attached a presentation that introduces some of my ideas at a high 
level. I think some people have seen this already, but it's probably a 
good time to share with a broader audience. I’ve also been doing some 
prototyping work exploring an architecture that addresses both of these 
points in the context of a 1-0 implementation. I have a more complete 
python version and a less complete (but improving) C version, both 
currently hosted at github:

https://github.com/rhs/amqp (python prototype)
https://github.com/rhs/amp (C prototype)

Note that the github location is purely for convenience of sharing and 
backup until I have something that will make sense to introduce to qpid.

As for how to incorporate this into qpid, I think if we want to become a 
ubiquitous protocol implementation, then the transport has to be an 
independently available component that is easily embeddable within third 
party apps that wish to integrate with messaging. As such it needs to 
have as few dependencies as possible and a very narrow and well 
considered set of interfaces. Given this, the natural way to develop 
this would be as an independent sub project within qpid, so that from 
the start we are forced to observe good packaging and dependency 
practices and consider our interfaces carefully.

--Rafael

Re: Qpid and AMQP 1-0: Plans?

[Robert Godfrey <ro...@gmail.com>]

On 24 March 2011 14:47, Rafael Schloming <ra...@redhat.com> wrote:

> On 03/24/2011 09:42 AM, Robert Godfrey wrote:
>
> I think one of the most important factors to consider is how we build out
> protocol support in a way that avoids some of the significant
> maintainability issues we’ve had with our previous approach. It’s pretty
> clear that one of our primary pain points has been with client support. The
> practice of transcoding wholesale from one client language into another has
> resulted in a significant number of write-only implementations that have
> proven impractical to maintain.
>
> IMHO, one of the key elements of our 1-0 strategy going forward has to be
> to share the minimum possible number of protocol implementations between as
> many different client and broker environments as possible. This goal has in
> fact been a key principle in the design of the 1-0 version of the protocol.
> Its symmetry and layering is specifically designed to allow for the majority
> of the protocol machinery to be encapsulated within a single transport layer
> with a simple interface that can be embedded within a wide variety of
> environments.
>
>
I agree with this.  The idea of a common transport layer is one that has
been raised before and which I think had broad support.  To me, the primary
goal of the Apache Qpid Project must be to aim to be the de facto standard
implementation of AMQP.  In order to achieve this goal we need to be able to
provide client APIs across as many languages as possible (and across OS and
machine architectures as well).  We can already see that our current
approach to this is not sustainable - we’ve had suggestions that we should
drop a number of client API artefacts from the 0.10 release as they are no
longer able to be maintained.

Having said that I think we need a little more clarity in defining what we
mean by a “common transport layer”.  Firstly, at what level do you see the
API here - is it simply an implementation of the AMQP 1-0 protocol, or are
we aiming at a higher abstraction that will allow us to have implementations
of the “transport layer” which use AMQP 0-10, 0-9-1 etc. as their protocol?
Secondly, if we are looking to share the transport layer between “clients”
and “brokers” it will need to be designed in such a way that the very
different programming models (by which I really mean threading models) work.

Lastly, and I guess most crucially, how do we see this work being
incorporated into our current Qpid implementations?

-- Rob



> --Rafael
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>

Re: Qpid and AMQP 1-0: Plans?

[Rafael Schloming <ra...@redhat.com>]

On 03/24/2011 09:42 AM, Robert Godfrey wrote:
> All,
>
> As we are now reaching the end of the 0.10 release process, I think it's
> probably an opportune moment to look at the goals of the Qpid project for
> the rest of 2011 and beyond.
>
> As many of you are no doubt aware, I have been spending the majority of my
> time over the last years on the AMQP 1-0 standard, which we can shortly
> expect to be released in its final form. In advance of this I think we need
> to be planning on how we are going to adapt Qpid so that AMQP 1-0 becomes
> its primary protocol.
>
> What are people's views about how we approach this?
>
> Cheers,
> Rob
>

I think one of the most important factors to consider is how we build 
out protocol support in a way that avoids some of the significant 
maintainability issues we’ve had with our previous approach. It’s pretty 
clear that one of our primary pain points has been with client support. 
The practice of transcoding wholesale from one client language into 
another has resulted in a significant number of write-only 
implementations that have proven impractical to maintain.

IMHO, one of the key elements of our 1-0 strategy going forward has to 
be to share the minimum possible number of protocol implementations 
between as many different client and broker environments as possible. 
This goal has in fact been a key principle in the design of the 1-0 
version of the protocol. Its symmetry and layering is specifically 
designed to allow for the majority of the protocol machinery to be 
encapsulated within a single transport layer with a simple interface 
that can be embedded within a wide variety of environments.

--Rafael

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org