You are viewing a plain text version of this content. The canonical link for it is here.
Posted to proton@qpid.apache.org by dcjohns41 <da...@honeywell.com> on 2014/05/09 17:41:56 UTC
Re: Minimizing the Proton Engine/Messenger RAM Footprint for
embedded devices
Thanks for the various suggestions on capturing the memory allocation. We
started the process and have found several areas that seem to be large RAM
users. Proton-C_MemoryAllocation.xlsx
<http://qpid.2158936.n2.nabble.com/file/n7607980/Proton-C_MemoryAllocation.xlsx>
The attached file is a list of the main startup allocation. In the
rightmost column, we have highlighted any allocation that is larger in size
in red. As you can see there are several 16KByte chunks of RAM allocated
(object.c /pni_map_allocate) as well as multiple 960 byte chunks(over 24
occurrences for approx. 24Kbytes)(pn_data.c). Also the pn_buffer and
pn_dispatcher functions each allocate 4Kbytes. This exceeded 75Kbytes prior
to our system being able to attempt communication setup.
So this drives a few questions as to being able to minimize these
allocations. The multiple 960 byte allocations and the 32Kbyte allocations
seem to be the big hitters.
Any hints from the mailing list on reducing these structures?
Dave
--
View this message in context: http://qpid.2158936.n2.nabble.com/Minimizing-the-Proton-Engine-Messenger-RAM-Footprint-for-embedded-devices-tp7607409p7607980.html
Sent from the Apache Qpid Proton mailing list archive at Nabble.com.
Re: Minimizing the Proton Engine/Messenger RAM Footprint for
embedded devices
Posted by Andrew Stitcher <as...@redhat.com>.
David,
Somewhat tangential to you original line of experiment: I've made a
couple of changes to the trunk version of proton in the last few days
that should completely eliminate static use of the .data section for
proton. There were a bunch of things that I've made "const" and so have
moved to .rodata in a static build and so should go into ROM with the
program and not RAM any longer.
This may save up to around 30-40k of data space depending on what gets
statically linked into your code. I don't know if any of your private
modifications will have preempted these changes so I suggest you give
trunk another go and see how much extra headroom you have now.
Just to be clear: these changes don't affect the dynamic allocation
behaviour of proton at all, just what is statically allocated before the
program even runs.
Andrew
RE: Minimizing the Proton Engine/Messenger RAM Footprint for
embedded devices
Posted by dcjohns41 <da...@honeywell.com>.
Rafael,
My team has integrated the trunk version of the Proton-c library. We were running the 0.6 release and it took a little time to bring in all the updates. Thanks for making an update so quickly.
The memory allocation is better in this new version-the 980 byte increments have been minimized. The next layer of the onion though is shown below:
- Two buffers in the transport.c/pn_transport_initialize for input/output at 16Kbytes each.
- 4Kbytes in buffer.c
- 4Kbytes in dispatcher.c
- 5.8Kbytes in multiple 832byte structures in codec.c (trace caught 7 allocations of this size before we ran out of memory)
The input/output buffers defaulting to 16 entries of 1024 each seems appropriate for a full fledged client, but for a small one if we made it much smaller would it affect many of the other messaging functions? For instance if we limited it to 2 entries.
Thank you for your continued help in this area.
Dave
From: Rafael Schloming-3 [via Qpid] [mailto:ml-node+s2158936n7608057h1@n2.nabble.com]
Sent: Monday, May 12, 2014 4:04 PM
To: Johnson, David (MN10)
Subject: Re: Minimizing the Proton Engine/Messenger RAM Footprint for embedded devices
I've checked in a few easy tweaks that should help some of the worst
offenders in your list. I don't know close it will get you to your goal,
but it would be helpful if you could rerun your test against the latest
code on trunk.
--Rafael
On Fri, May 9, 2014 at 11:41 AM, dcjohns41 <[hidden email]</user/SendEmail.jtp?type=node&node=7608057&i=0>>wrote:
> Thanks for the various suggestions on capturing the memory allocation. We
> started the process and have found several areas that seem to be large RAM
> users. Proton-C_MemoryAllocation.xlsx
> <
> http://qpid.2158936.n2.nabble.com/file/n7607980/Proton-C_MemoryAllocation.xlsx
> >
>
> The attached file is a list of the main startup allocation. In the
> rightmost column, we have highlighted any allocation that is larger in size
> in red. As you can see there are several 16KByte chunks of RAM allocated
> (object.c /pni_map_allocate) as well as multiple 960 byte chunks(over 24
> occurrences for approx. 24Kbytes)(pn_data.c). Also the pn_buffer and
> pn_dispatcher functions each allocate 4Kbytes. This exceeded 75Kbytes
> prior
> to our system being able to attempt communication setup.
>
> So this drives a few questions as to being able to minimize these
> allocations. The multiple 960 byte allocations and the 32Kbyte allocations
> seem to be the big hitters.
>
> Any hints from the mailing list on reducing these structures?
>
> Dave
>
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Minimizing-the-Proton-Engine-Messenger-RAM-Footprint-for-embedded-devices-tp7607409p7607980.html
> Sent from the Apache Qpid Proton mailing list archive at Nabble.com.
>
________________________________
If you reply to this email, your message will be added to the discussion below:
http://qpid.2158936.n2.nabble.com/Minimizing-the-Proton-Engine-Messenger-RAM-Footprint-for-embedded-devices-tp7607409p7608057.html
To unsubscribe from Minimizing the Proton Engine/Messenger RAM Footprint for embedded devices, click here<http://qpid.2158936.n2.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7607409&code=ZGF2aWQuam9obnNvbjNAaG9uZXl3ZWxsLmNvbXw3NjA3NDA5fDIwNzc4NzgwMTA=>.
NAML<http://qpid.2158936.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://qpid.2158936.n2.nabble.com/Minimizing-the-Proton-Engine-Messenger-RAM-Footprint-for-embedded-devices-tp7607409p7608231.html
Sent from the Apache Qpid Proton mailing list archive at Nabble.com.
RE: Minimizing the Proton Engine/Messenger RAM Footprint for
embedded devices
Posted by dcjohns41 <da...@honeywell.com>.
We will give it a try and let you know..
Thanks
Dave
From: Rafael Schloming-3 [via Qpid] [mailto:ml-node+s2158936n7608057h1@n2.nabble.com]
Sent: Monday, May 12, 2014 4:04 PM
To: Johnson, David (MN10)
Subject: Re: Minimizing the Proton Engine/Messenger RAM Footprint for embedded devices
I've checked in a few easy tweaks that should help some of the worst
offenders in your list. I don't know close it will get you to your goal,
but it would be helpful if you could rerun your test against the latest
code on trunk.
--Rafael
On Fri, May 9, 2014 at 11:41 AM, dcjohns41 <[hidden email]</user/SendEmail.jtp?type=node&node=7608057&i=0>>wrote:
> Thanks for the various suggestions on capturing the memory allocation. We
> started the process and have found several areas that seem to be large RAM
> users. Proton-C_MemoryAllocation.xlsx
> <
> http://qpid.2158936.n2.nabble.com/file/n7607980/Proton-C_MemoryAllocation.xlsx
> >
>
> The attached file is a list of the main startup allocation. In the
> rightmost column, we have highlighted any allocation that is larger in size
> in red. As you can see there are several 16KByte chunks of RAM allocated
> (object.c /pni_map_allocate) as well as multiple 960 byte chunks(over 24
> occurrences for approx. 24Kbytes)(pn_data.c). Also the pn_buffer and
> pn_dispatcher functions each allocate 4Kbytes. This exceeded 75Kbytes
> prior
> to our system being able to attempt communication setup.
>
> So this drives a few questions as to being able to minimize these
> allocations. The multiple 960 byte allocations and the 32Kbyte allocations
> seem to be the big hitters.
>
> Any hints from the mailing list on reducing these structures?
>
> Dave
>
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Minimizing-the-Proton-Engine-Messenger-RAM-Footprint-for-embedded-devices-tp7607409p7607980.html
> Sent from the Apache Qpid Proton mailing list archive at Nabble.com.
>
________________________________
If you reply to this email, your message will be added to the discussion below:
http://qpid.2158936.n2.nabble.com/Minimizing-the-Proton-Engine-Messenger-RAM-Footprint-for-embedded-devices-tp7607409p7608057.html
To unsubscribe from Minimizing the Proton Engine/Messenger RAM Footprint for embedded devices, click here<http://qpid.2158936.n2.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7607409&code=ZGF2aWQuam9obnNvbjNAaG9uZXl3ZWxsLmNvbXw3NjA3NDA5fDIwNzc4NzgwMTA=>.
NAML<http://qpid.2158936.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://qpid.2158936.n2.nabble.com/Minimizing-the-Proton-Engine-Messenger-RAM-Footprint-for-embedded-devices-tp7607409p7608058.html
Sent from the Apache Qpid Proton mailing list archive at Nabble.com.
Re: Minimizing the Proton Engine/Messenger RAM Footprint for embedded devices
Posted by Rafael Schloming <rh...@alum.mit.edu>.
I've checked in a few easy tweaks that should help some of the worst
offenders in your list. I don't know close it will get you to your goal,
but it would be helpful if you could rerun your test against the latest
code on trunk.
--Rafael
On Fri, May 9, 2014 at 11:41 AM, dcjohns41 <da...@honeywell.com>wrote:
> Thanks for the various suggestions on capturing the memory allocation. We
> started the process and have found several areas that seem to be large RAM
> users. Proton-C_MemoryAllocation.xlsx
> <
> http://qpid.2158936.n2.nabble.com/file/n7607980/Proton-C_MemoryAllocation.xlsx
> >
>
> The attached file is a list of the main startup allocation. In the
> rightmost column, we have highlighted any allocation that is larger in size
> in red. As you can see there are several 16KByte chunks of RAM allocated
> (object.c /pni_map_allocate) as well as multiple 960 byte chunks(over 24
> occurrences for approx. 24Kbytes)(pn_data.c). Also the pn_buffer and
> pn_dispatcher functions each allocate 4Kbytes. This exceeded 75Kbytes
> prior
> to our system being able to attempt communication setup.
>
> So this drives a few questions as to being able to minimize these
> allocations. The multiple 960 byte allocations and the 32Kbyte allocations
> seem to be the big hitters.
>
> Any hints from the mailing list on reducing these structures?
>
> Dave
>
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Minimizing-the-Proton-Engine-Messenger-RAM-Footprint-for-embedded-devices-tp7607409p7607980.html
> Sent from the Apache Qpid Proton mailing list archive at Nabble.com.
>