Proton Embedded Proposal

["Chris Busbey (MS OPEN TECH)" <ch...@microsoft.com>]

Hi there,

We have recently had the opportunity to utilize Proton on embedded devices to talk to Azure.  The class of embedded devices we are working with require some strict design constraints on the AMQP library we can use:


*         Library must be AMQP 1.0 compatible

*         Library must run on embedded devices running a micro RTOS, TCP/IP and SSL

*         Library should avoid heap fragmentation

*         Library RAM use must be less than 25K

*         Library Flash use must be less than 75k

These requirements are not met by Proton out of the box, but we do think they align with the Proton roadmap to support this class of devices.

In order to accomplish these goals, we are proposing to alter some of the underlying Proton-C architecture.  Here is an overview of the changes we are proposing:

Target Micro RTOS, TCP/IP, and SSL - Introduce a porting layer to provide abstractions for required RTOS constructs.  These constructs are limited to basic constructs- threads, queues, semaphores.  Some changes may be necessary to adapt the socket and network interface to RTOS requirements and improve portability across RTOS and network stacks.  Devices running without an RTOS are not supported.  A porting layer provides abstractions for the required Embedded SSL constructs.

Memory Allocation and Management - To make more efficient use of the available memory and avoid fragmentation, calls to malloc() may be replaced by specially tailored memory allocation routines and/or block pools.

Limited Protocol Features - In order to meet RAM and flash usage targets, protocol features can be limited.  The allowable limits are set to assure the implementation remains 1.0 compatible.  These limits will be documented.

Initially, we would target the following platforms and libraries:

RTOS:
*         ThreadX
*         FreeRTOS

TCP/IP stacks:
*         NetX
*         LwIP

SSL Libraries:
*         Mocana
*         Matrix

Although we would prefer to contribute these changes back to the Proton-C source tree, we think these changes would radically alter the Proton-C codebase as it stands now.  A better solution may be to introduce a new class of Proton- call it Proton Embedded or Proton-e.  We have already kicked off this work and would like to contribute these changes back to the community, most likely in a branch so that the community can review the code thoroughly before deciding on a final home.

Would there be interest in this work added to the Proton project?

Chris Busbey
Senior Program Manager
Microsoft Open Technologies, Inc.
A subsidiary of Microsoft Corporation

Re: Proton Embedded Proposal

[Rafael Schloming <rh...@alum.mit.edu>]

Hi there,

First off, thanks for bringing this to the list. I would love to see Proton
have better support for embedded environments, so any contributions you
could make in that area would be very welcome.

I've got a bunch of more detailed questions/comments inline, so please read
on...

On Wed, Nov 5, 2014 at 5:00 PM, Chris Busbey (MS OPEN TECH) <
chbusbey@microsoft.com> wrote:

> Hi there,
>
> We have recently had the opportunity to utilize Proton on embedded devices
> to talk to Azure.  The class of embedded devices we are working with
> require some strict design constraints on the AMQP library we can use:
>
>
> *         Library must be AMQP 1.0 compatible
>
> *         Library must run on embedded devices running a micro RTOS,
> TCP/IP and SSL
>
> *         Library should avoid heap fragmentation
>
> *         Library RAM use must be less than 25K
>
> *         Library Flash use must be less than 75k
>
> These requirements are not met by Proton out of the box, but we do think
> they align with the Proton roadmap to support this class of devices.
>

The core engine and events portion of proton doesn't make any use of
threads or sockets. Assuming your use is limited to this portion of proton,
I would think the only issue you would run into would be memory related. Is
this accurate? Can you provide anymore detail about your usage and what
sort of issues you ran into?


>
> In order to accomplish these goals, we are proposing to alter some of the
> underlying Proton-C architecture.  Here is an overview of the changes we
> are proposing:
>
> Target Micro RTOS, TCP/IP, and SSL - Introduce a porting layer to provide
> abstractions for required RTOS constructs.  These constructs are limited to
> basic constructs- threads, queues, semaphores.  Some changes may be
> necessary to adapt the socket and network interface to RTOS requirements
> and improve portability across RTOS and network stacks.  Devices running
> without an RTOS are not supported.  A porting layer provides abstractions
> for the required Embedded SSL constructs.
>

Do you have a rough sketch of what this porting layer would look like?


>
> Memory Allocation and Management - To make more efficient use of the
> available memory and avoid fragmentation, calls to malloc() may be replaced
> by specially tailored memory allocation routines and/or block pools.
>

This makes sense and I think some of that work would probably be useful in
non embedded scenarios also.


>
> Limited Protocol Features - In order to meet RAM and flash usage targets,
> protocol features can be limited.  The allowable limits are set to assure
> the implementation remains 1.0 compatible.  These limits will be documented.
>

This one is an interesting one. To the greatest extent possible I would
like to avoid duplicate implementation of the protocol logic itself.
Supporting alternative network stacks, alternative SSL implementations, and
alternative threading/control flow models are all things we've tried to
consider and IMHO would be reasonably doable with the current codebase. For
this one it seems like ideally it would be nice to be able to have a single
protocol implementation that would scale up/down as needed. What sort of
limitations are you considering here?


> Initially, we would target the following platforms and libraries:
>
> RTOS:
> *         ThreadX
> *         FreeRTOS
>
> TCP/IP stacks:
> *         NetX
> *         LwIP
>
> SSL Libraries:
> *         Mocana
> *         Matrix
>
> Although we would prefer to contribute these changes back to the Proton-C
> source tree, we think these changes would radically alter the Proton-C
> codebase as it stands now.  A better solution may be to introduce a new
> class of Proton- call it Proton Embedded or Proton-e.  We have already
> kicked off this work and would like to contribute these changes back to the
> community, most likely in a branch so that the community can review the
> code thoroughly before deciding on a final home.
>

Can you give me a bit more detail about how you think the changes would
alter the proton-c codebase?

To be honest I'm a little nervous at the thought of having an entirely new
flavor of proton itself. There's a fair bit of overhead in dealing with the
two existing flavors, so even if it would involve some more disruptive
changes, I have a strong bias to keep things more compact and manageable.
The various parts of proton are designed to be somewhat replaceable in
order to accommodate a wide range of uses, e.g. you should be able to write
an entirely new driver for a non-socket based network stack, and even if
you do need a specialized/limited engine implementation, it can still use
the same codec and generate the same events that the current engine does,
thereby making it much more compatible with anything written against the
current engine.

In summary what I'd really like to understand is if we can address the
embedded scenario by swapping out pieces rather than providing an entirely
different "flavor". I think the benefit of this is that you could pick the
tradeoffs you need for your specific application, e.g. you could choose to
use a bandwidth-optimized codec + fully capable engine if you have say a
more powerful device on a low bandwidth link. If this isn't possible I'd
like to understand what are the limitations of the current codebase that
prevent this. I suspect some up front work to address those limitations
would likely pay off quickly relative to the long term overhead of having a
whole new flavor to manage.

--Rafael