You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Rick van Rein <ri...@openfortress.nl> on 2017/06/21 18:50:58 UTC
Revisiting Qpid Proton / Erlang
Hello,
I have been exploring options of using AMQP in connection with various
languages. I strongly prefer 1.0 because it can allow cross-domain
messaging, so users could push MIME-typed content after passing
authentication & authorisation.
Proton C seems to hold good cards in terms of language support. But to
me, Erlang is a great amiss in the list. Are people still working on
its integration?
This is not a pet language (I hope). Erlang offers a platform for
building rock-solid solutions, which has always had a clear niche of
applications. These rock-solid solutions are clear casses to benefit
from AMQP.
I'm wondering if we'll have to go for < 1.0 by adopting RabbitMQ as that
is integrated with many more languages, especially with Erlang of course
(it's written in it). Or whether to believe that Qpid has the advantage
of the newcomer that can jump ahead. The thing is...
1. Qpid Dispatch Router is an attractive and novel use of 1.0
2. RabbitMQ could serve as a gateway 1.0 server but AFAIK not in the
role of a 1.0 client to Qpid Dispatch Router
3. Java is not acceptable to all who are running a server (and we'd
like to avoid it for that reason)
...that it does not seem to be possible to enter the 1.0 World from
Erlang's robustness-supporting platform -- and entering 1.0 is just what
the Qpid aims to support.
I've had a look at how difficult making the integration would be, of
course. The Erlang side seems doable, given the Port concept to
communicate with C nodes, and the ei interface for sequencing Erlang
binary data. On the Qpid side, the Electron approach matches well with
the Erlang concept of many small processes (as in Go). But I did not
find API specifications on the Qpid side, only examples -- and that's
insufficient to build a translation for anyone but the original designers...
Seeing the earlier attention to this link to Erlang, I would assume that
more people are interested in it being supported, probably following a
similar strand of reasoning as I am above. What can we do to get that
into motion?
Thanks,
Rick van Rein
InternetWide.org / ARPA2.net
P.S.
Here are two use cases for AMQP that we have in mind:
http://reservoir.arpa2.net
http://internetwide.org/blog/2017/05/03/building-with-blocks.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Revisiting Qpid Proton / Erlang
Posted by Alan Conway <ac...@redhat.com>.
On Wed, 2017-06-21 at 23:41 +0200, Rick van Rein wrote:
> Hi,
>
> Thank you all for taking my remarks seriously.
>
>
> > [Rob Godfrey]
> > From Rick's e-mail I get the impression that he feels there is insufficient
> > documentation in order to do the work without support from the Electron(?)
> > authors:
>
> If I've seen everything (broker/sender/receiver code) then it would be difficult as a basis, yes. APIs tend to harbour "obvious" assumptions, even when they are well-designed, and I would not dare to barge in based on working code -- however useful code is as a proof of concept!
>
That's everything for now (including the API doc of course) but I would
like to expand on it. I am steeped in assumptions from working on Qpid
projects so if you can use your fresh perspective to point to specific
gaps and questions I'd like to fill them in. I'm sure there are real
gaps in the API too which you can help uncover as you look at it from a
new angle.
> > [Alan Conway]
> > I wrote the electron API for Go, and I'd be happy to discuss it further
> > with you and offer thoughts on API design, but I don't know any Erlang
> > (although I gather Go borrows some ideas from it)
>
> Go borrowed precisely those things that made you pitch Electron
> (which I think is a great idea by the way), namely:
>
> - OS threads only used to occupy multiple CPU cores
> - language-own lightweight threading
> - heap-based contexts for many small processes
and language support for CSP to shift the design focus of concurrent
programming from shared data to co-operating functions: https://blog.go
lang.org/share-memory-by-communicating
> Erlang also does a few things better IMHO, making it such a great
> language for AMQP applications:
>
> - near-trivial distribution of work over a cluster
> - transparant communication on its paranoia-encapsulating platform(OTS)
> - garbage collection per mini-proces (for soft realtime responsiveness)
>
> > Inversion does not suit Go (or Erlang I suspect) so I wrote the
> > qpid.apache.org/electron API to be coroutine-friendly, asynchronous,
> > but "imperative".
>
> Erlang uses functional programming, but it is pragmatic in doing so:
>
> - it sort-of stores data locally in each mini-proces
> - it feels a bit imperative (one assignment per variable per context)
> - it actually has an imperative dictionary for each mini-proces
> - communication between mini-processes adds to their incoming "mailbox"
>
> The dirty / imperative aspects are often contained in generic
> design patterns in the OTP platform. This is where the rich
> cruft of concurrency, recovery from faults and so on hides,
> leaving the user with a reasonably clean surface to work on.
>
> > I wrote the qpid.apache.org/proton package while learning Go and
> > following the model of the other bindings, but if I was doing it again
> > I would write electron directly in terms of proton-C - all the Go
> > developer interest is in electron.
>
> I would expect the same to happen with Erlang.
>
> > I have a strong interest in extending the "imperative" or "uninverted"
> > API approach to other languages - it is ideal for coroutine languages,
> > but even with traditional threads I think it is much easier to code
> > "moderately concurrent" apps this way.
>
> Yes, agreed! Processes, like objects, help to structure software.
> This is something I love about Go and Erlang.
> (Also, CSP-style message passing is simpler than pthreads. Well,
> anything is simpler than pthreads I suppose?)
You can do CSP-like programming with pthreads but its a lot of heavy
lifting. The qpidd and dispatch servers have always been structured as
a "sequential context" per connection, and added some ad-hoc extra
contexts (qpidd called them "pollable queues")
In the current C/C++ APIs we're working towards some core support for a
generic "work_queue" abstraction - it is already used internally for
per-connection serialization and by making it available to users it
makes a C++ broker much easier to write (examples coming shortly!)
This is not directly relevant to Go or Erlang yet, but I'm hoping
to work up solid CSP-friendly APIs using the native Go/Erlang tools,
and the carry ideas back to new/extended C++, ruby and python APIs
using work_queues, futures or other emerging tools in those langauges.
> can't write any Erlang but I'd be very happy discuss/review API
> > proposals, donate my experience from Go and Ruby, and give pointers on
> > the C side of things. Once there is some code I can help with getting
> > it set up as part of the Qpid project.
>
> Thank you!
>
> There are two major sides to Erlang that are involved in porting I suppose,
>
> http://erlang.org/doc/reference_manual/ports.html
>
> Ports communicate with Erlang mini-processes from C or Go or...
>
> http://erlang.org/doc/man/ei.html
>
> The ei API serialises data passed into / out of Erlang over a Port.
>
> The language is remarkably easy to learn, by the way... given conceptual
> understanding of functional and concurrent programming.
I would love to dig into it but don't have enough spare time to learn
erlang right now. If you can ask specific questions about the electron
doc/design that might be the best way to suck me in. I am itching to be
sucked in...
>
>
> Good, time for some soul-searching on my end... like, where to find the time
> (or man power) for this useful diversion :-S
>
>
> Thanks,
> -Rick
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Revisiting Qpid Proton / Erlang
Posted by Rick van Rein <ri...@openfortress.nl>.
Hi,
Thank you all for taking my remarks seriously.
> [Rob Godfrey]
> From Rick's e-mail I get the impression that he feels there is insufficient
> documentation in order to do the work without support from the Electron(?)
> authors:
If I've seen everything (broker/sender/receiver code) then it would be difficult as a basis, yes. APIs tend to harbour "obvious" assumptions, even when they are well-designed, and I would not dare to barge in based on working code -- however useful code is as a proof of concept!
> [Alan Conway]
> I wrote the electron API for Go, and I'd be happy to discuss it further
> with you and offer thoughts on API design, but I don't know any Erlang
> (although I gather Go borrows some ideas from it)
Go borrowed precisely those things that made you pitch Electron
(which I think is a great idea by the way), namely:
- OS threads only used to occupy multiple CPU cores
- language-own lightweight threading
- heap-based contexts for many small processes
Erlang also does a few things better IMHO, making it such a great
language for AMQP applications:
- near-trivial distribution of work over a cluster
- transparant communication on its paranoia-encapsulating platform(OTS)
- garbage collection per mini-proces (for soft realtime responsiveness)
> Inversion does not suit Go (or Erlang I suspect) so I wrote the
> qpid.apache.org/electron API to be coroutine-friendly, asynchronous,
> but "imperative".
Erlang uses functional programming, but it is pragmatic in doing so:
- it sort-of stores data locally in each mini-proces
- it feels a bit imperative (one assignment per variable per context)
- it actually has an imperative dictionary for each mini-proces
- communication between mini-processes adds to their incoming "mailbox"
The dirty / imperative aspects are often contained in generic
design patterns in the OTP platform. This is where the rich
cruft of concurrency, recovery from faults and so on hides,
leaving the user with a reasonably clean surface to work on.
> I wrote the qpid.apache.org/proton package while learning Go and
> following the model of the other bindings, but if I was doing it again
> I would write electron directly in terms of proton-C - all the Go
> developer interest is in electron.
I would expect the same to happen with Erlang.
> I have a strong interest in extending the "imperative" or "uninverted"
> API approach to other languages - it is ideal for coroutine languages,
> but even with traditional threads I think it is much easier to code
> "moderately concurrent" apps this way.
Yes, agreed! Processes, like objects, help to structure software.
This is something I love about Go and Erlang.
(Also, CSP-style message passing is simpler than pthreads. Well,
anything is simpler than pthreads I suppose?)
> I can't write any Erlang but I'd be very happy discuss/review API
> proposals, donate my experience from Go and Ruby, and give pointers on
> the C side of things. Once there is some code I can help with getting
> it set up as part of the Qpid project.
Thank you!
There are two major sides to Erlang that are involved in porting I suppose,
http://erlang.org/doc/reference_manual/ports.html
Ports communicate with Erlang mini-processes from C or Go or...
http://erlang.org/doc/man/ei.html
The ei API serialises data passed into / out of Erlang over a Port.
The language is remarkably easy to learn, by the way... given conceptual
understanding of functional and concurrent programming.
Good, time for some soul-searching on my end... like, where to find the time
(or man power) for this useful diversion :-S
Thanks,
-Rick
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Revisiting Qpid Proton / Erlang
Posted by Alan Conway <ac...@redhat.com>.
On Wed, 2017-06-21 at 20:50 +0200, Rick van Rein wrote:
> Hello,
>
> I have been exploring options of using AMQP in connection with various
> languages. I strongly prefer 1.0 because it can allow cross-domain
> messaging, so users could push MIME-typed content after passing
> authentication & authorisation.
>
> Proton C seems to hold good cards in terms of language support. But to
> me, Erlang is a great amiss in the list. Are people still working on
> its integration?
>
> This is not a pet language (I hope). Erlang offers a platform for
> building rock-solid solutions, which has always had a clear niche of
> applications. These rock-solid solutions are clear casses to benefit
> from AMQP.
>
> I'm wondering if we'll have to go for < 1.0 by adopting RabbitMQ as that
> is integrated with many more languages, especially with Erlang of course
> (it's written in it). Or whether to believe that Qpid has the advantage
> of the newcomer that can jump ahead. The thing is...
>
> 1. Qpid Dispatch Router is an attractive and novel use of 1.0
> 2. RabbitMQ could serve as a gateway 1.0 server but AFAIK not in the
> role of a 1.0 client to Qpid Dispatch Router
> 3. Java is not acceptable to all who are running a server (and we'd
> like to avoid it for that reason)
>
> ...that it does not seem to be possible to enter the 1.0 World from
> Erlang's robustness-supporting platform -- and entering 1.0 is just what
> the Qpid aims to support.
>
> I've had a look at how difficult making the integration would be, of
> course. The Erlang side seems doable, given the Port concept to
> communicate with C nodes, and the ei interface for sequencing Erlang
> binary data. On the Qpid side, the Electron approach matches well with
> the Erlang concept of many small processes (as in Go). But I did not
> find API specifications on the Qpid side, only examples -- and that's
> insufficient to build a translation for anyone but the original designers...
I wrote the electron API for Go, and I'd be happy to discuss it further
with you and offer thoughts on API design, but I don't know any Erlang
(although I gather Go borrows some ideas from it)
I first wrote the qpid.apache.org/proton package which is a very thin
layer over the C API - in fact it's mostly code-generated. It is an
inversion-of-control API, like the other proton language bindings.
Inversion does not suit Go (or Erlang I suspect) so I wrote the
qpid.apache.org/electron API to be coroutine-friendly, asynchronous,
but "imperative". It aims to stay close to the AMQP protocol concepts
(which keeps it fairly close to the other proton APIs) but it
introduces some new asynchronous tactics to replace the upside-down
event loop.
I wrote the qpid.apache.org/proton package while learning Go and
following the model of the other bindings, but if I was doing it again
I would write electron directly in terms of proton-C - all the Go
developer interest is in electron. I think it is worth wrapping the
proton-C library, it hides a lot of AMQP complexity. I wouldn't mind
writing a pure Go package someday but so far people seem happy with the
hybrid.
I have a strong interest in extending the "imperative" or "uninverted"
API approach to other languages - it is ideal for coroutine languages,
but even with traditional threads I think it is much easier to code
"moderately concurrent" apps this way. I'm planning to try to migrate
some of the ideas to Ruby to solve problems I've had with the inverted
approach when integrating AMQP into threaded Ruby applications.
> Seeing the earlier attention to this link to Erlang, I would assume that
> more people are interested in it being supported, probably following a
> similar strand of reasoning as I am above. What can we do to get that
> into motion?
>
I can't write any Erlang but I'd be very happy discuss/review API
proposals, donate my experience from Go and Ruby, and give pointers on
the C side of things. Once there is some code I can help with getting
it set up as part of the Qpid project.
Cheers,
Alan.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Revisiting Qpid Proton / Erlang
Posted by Rob Godfrey <ro...@gmail.com>.
On 21 June 2017 at 21:40, Gordon Sim <gs...@redhat.com> wrote:
> On 21/06/17 20:37, Gordon Sim wrote:
>
>> On 21/06/17 19:50, Rick van Rein wrote:
>>
>>> I have been exploring options of using AMQP in connection with various
>>> languages. I strongly prefer 1.0 because it can allow cross-domain
>>> messaging, so users could push MIME-typed content after passing
>>> authentication & authorisation.
>>>
>>> Proton C seems to hold good cards in terms of language support. But to
>>> me, Erlang is a great amiss in the list. Are people still working on
>>> its integration?
>>>
>>> This is not a pet language (I hope). Erlang offers a platform for
>>> building rock-solid solutions, which has always had a clear niche of
>>> applications. These rock-solid solutions are clear casses to benefit
>>> from AMQP.
>>>
>>
>> I agree that an AMQP 1.0 library for Erlang would be very valuable. I
>> don't have any experience with that language myself. The rabbitmq team
>> appear to be working on something which would be worth a look as they
>> obviously have a lot of expertise in the area:
>>
>> https://github.com/rabbitmq/rabbitmq-amqp1.0-client
>>
>
> I should clarify here that I do think it would be great to have an erlang
> client in Qpid, but (a) I think that would require new contributors as
> there is already a lot of work to do and (b) joining an existing open
> source initiative might be preferable to starting something new.
>
>
>
I agree - it would be fantastic to see an Erlang client in Qpid, but as
Gordon says in order for it to be sustainable in Qpid it needs people here
who can contribute and maintain it. If Rick, or others who read this, are
able to commit some of their time to it that would be fantastic. Obviously
the Qpid community is always keen to see new contributors and would welcome
people aboard.
From Rick's e-mail I get the impression that he feels there is insufficient
documentation in order to do the work without support from the Electron(?)
authors:
> I've had a look at how difficult making the integration would be, of
> course. The Erlang side seems doable, given the Port concept to
> communicate with C nodes, and the ei interface for sequencing Erlang
> binary data. On the Qpid side, the Electron approach matches well with
> the Erlang concept of many small processes (as in Go). But I did not
> find API specifications on the Qpid side, only examples -- and that's
> insufficient to build a translation for anyone but the original
> designers...
RIck - is my understanding correct? If so would it be possible to explain
more which areas need more documentation, or possibly just ask questions
directly on parts of the API that are unclear (to be clear here, I am
completely unfamiliar with the Electron side of things so I am totally
unable to help here). Hopefully those more familiar with the relevant
parts of the Qpid codebase could help out here.
Cheers,
Rob
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Revisiting Qpid Proton / Erlang
Posted by Gordon Sim <gs...@redhat.com>.
On 21/06/17 20:37, Gordon Sim wrote:
> On 21/06/17 19:50, Rick van Rein wrote:
>> I have been exploring options of using AMQP in connection with various
>> languages. I strongly prefer 1.0 because it can allow cross-domain
>> messaging, so users could push MIME-typed content after passing
>> authentication & authorisation.
>>
>> Proton C seems to hold good cards in terms of language support. But to
>> me, Erlang is a great amiss in the list. Are people still working on
>> its integration?
>>
>> This is not a pet language (I hope). Erlang offers a platform for
>> building rock-solid solutions, which has always had a clear niche of
>> applications. These rock-solid solutions are clear casses to benefit
>> from AMQP.
>
> I agree that an AMQP 1.0 library for Erlang would be very valuable. I
> don't have any experience with that language myself. The rabbitmq team
> appear to be working on something which would be worth a look as they
> obviously have a lot of expertise in the area:
>
> https://github.com/rabbitmq/rabbitmq-amqp1.0-client
I should clarify here that I do think it would be great to have an
erlang client in Qpid, but (a) I think that would require new
contributors as there is already a lot of work to do and (b) joining an
existing open source initiative might be preferable to starting
something new.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Revisiting Qpid Proton / Erlang
Posted by Rick van Rein <ri...@openfortress.nl>.
Thanks Gordon!
> The rabbitmq team appear to be working on something which would be
> worth a look as they obviously have a lot of expertise in the area:
>
> https://github.com/rabbitmq/rabbitmq-amqp1.0-client
That is new to me. RabbitMQ might be an intermediate stepping stone.
Though Qpid on Erlang is still the holy grail IMHO :)
-Rick
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Revisiting Qpid Proton / Erlang
Posted by Gordon Sim <gs...@redhat.com>.
On 21/06/17 19:50, Rick van Rein wrote:
> I have been exploring options of using AMQP in connection with various
> languages. I strongly prefer 1.0 because it can allow cross-domain
> messaging, so users could push MIME-typed content after passing
> authentication & authorisation.
>
> Proton C seems to hold good cards in terms of language support. But to
> me, Erlang is a great amiss in the list. Are people still working on
> its integration?
>
> This is not a pet language (I hope). Erlang offers a platform for
> building rock-solid solutions, which has always had a clear niche of
> applications. These rock-solid solutions are clear casses to benefit
> from AMQP.
I agree that an AMQP 1.0 library for Erlang would be very valuable. I
don't have any experience with that language myself. The rabbitmq team
appear to be working on something which would be worth a look as they
obviously have a lot of expertise in the area:
https://github.com/rabbitmq/rabbitmq-amqp1.0-client
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org