You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mynewt.apache.org by "paul@wrada.com" <pa...@wrada.com> on 2016/06/21 19:31:48 UTC
split image to increase application storage space.
I’m working on split image feature and I think I just have one more major design issue to consider, and that is newt build related.
First, a summary for folks who are unaware of this effort. The goal is to create an application in two pieces to fit into two image banks such that one piece would contain the bluetooth stack and firmware update application and the other would contain the primary customer application (that’s the goal, but it would be defined generally to allow any split). These two are linked together with a special property that the upgrade app could run without the primary customer app but not vice versa. To give these names I call the independent one the AIIC (Application independent image component) and the customer app the ADIC (Application Dependent Image component). This would allow the following upgrade procedure with two flash images. At each step, there would always be a valid upgrade image loaded into the unit.
1. Erase the application image from the second image bank (still can recover since the upgrade image is valid)
2. Upload a new upgrade image to the second image bank (primary upgrade image still intact in case secondary fails)
3. Swap the upgrade images in the 1st and 2nd bank (if secondary doesn’t boot, we can always revert)
4. Load the new application into the 2nd bank (if this fails, we still have the upgraded in the primary)
5. Complete
This model allows a safe upgrade (because the upgrade image can always upgrade) while preserving more space for the application because the application doesn’t have to duplicate the bluetooth stack and associated upgrade code.
Consider this example, two 112k flash banks which we want to store a bluetooth application. Assume the following code sizes: Application size 32k, Bluetooth stack size 64k, Upgrade code size 16k. With two independent app images, each would be 112k filling the available sectors. However, if we split the image we would have an upgrade image of 80k and an app image of 32k (since it uses bluetooth and upgrade from the AIIC) leaving tons more space for sophisticated applications and more space for the upgrade as well.
We decided to create this split image as a pair that are linked together during the build process. There will be no dynamic bindings like SWI or function table or anything like that. The goal is not to separate the OS/stack from the app, but just to be a bit more efficient about duplicating the large components like bluetooth. Thus, when up are updating one of the split images, you MUST update the other. Splitting the image makes this “safe" as there will always be a bank filled with an image that is capable of performing an upgrade.
Anyway, that was the background, now on to my design question
The question I am investigating is how to formulate this in the newt build process. Here are two alternatives.
IDEA1: I was thinking that I would have a single target to produce both of these image components. The target would still have one bsp, one build profile, but I would define a new token for the target called loader=xxx. This would define the loader AIIC that would go into the first image bank, and if “loader" was defined, the app would be linked to run in the second image bank. Thus, if I were building a bluetooth app called foo, I would set loader=AIIC (contains upgrade and bluetooth) and then set app=foo. If “loader” was not defined, I would ensure the newt tool would do a complete build of foo and link it into the first image bank.
IDEA2: The alternative would be to have two separate targets. The AIDC target would have to reference the AIIC target and know that it was being built to be dependent on that bit. In this case, the AIIC target would be build as normal with no changes to the build system. Then the AIDC target would have to have a new tag that said that it was dependent on the other target. For example, in the app:foo target descriptor it would say split_image_dep=AIIC. This would cause newt to perform some special processing of the AIIC elf image, and ensure that we link the foo AIDC app to run out of the second image bank and reference common functions in the first image bank.
In either case, the application will be build as a split image and will contain two elf files. One file will be the Application Independent AIIC) part, and another will be the application dependent (ADIC) part. These will both get converted to image files by newt create-image.
Comments? My thoughts are that IDEA1 is the better of these two. It would allow newt to be split-image-aware during create-image and during download and debug. Any other ideas from someone more knowledgeable about newt?
Paul
Re: split image to increase application storage space.
Posted by Sterling Hughes <st...@apache.org>.
>
>
> Comments? My thoughts are that IDEA1 is the better of these two. It
> would allow newt to be split-image-aware during create-image and
> during download and debug. Any other ideas from someone more
> knowledgeable about newt?
>
+1
Re: split image to increase application storage space.
Posted by "paul@wrada.com" <pa...@wrada.com>.
David,
Thanks for the feedback. Our current design will directly link function
calls between the AIDC and AIIC. It would not be safe to upgrade the
application without upgrading the AIIC as well.
If we had to consider supporting upgrade of the AIDC without upgrading the
AIIC, that would be a significantly more complex design.
Paul
On 6/21/16, 1:12 PM, "David G. Simmons" <sa...@mac.com> wrote:
>+1 This is very similar to how we handled upgrading images on a previous
>IoT platform and it was very good at keeping customers from bricking
>their device. I also like that it allows for the upgrading of just the
>ADIC without touching a valid and functioning BLE Stack, etc. which makes
>it more robust in that there is a lower chance of intruducing a
>lower-level bug into the system simply by updating the application.
>
>dg
>
>> On Jun 21, 2016, at 3:31 PM, paul@wrada.com wrote:
>>
>> I’m working on split image feature and I think I just have one more
>>major design issue to consider, and that is newt build related.
>>
>> First, a summary for folks who are unaware of this effort. The goal is
>>to create an application in two pieces to fit into two image banks such
>>that one piece would contain the bluetooth stack and firmware update
>>application and the other would contain the primary customer application
>>(that’s the goal, but it would be defined generally to allow any split).
>> These two are linked together with a special property that the upgrade
>>app could run without the primary customer app but not vice versa. To
>>give these names I call the independent one the AIIC (Application
>>independent image component) and the customer app the ADIC (Application
>>Dependent Image component). This would allow the following upgrade
>>procedure with two flash images. At each step, there would always be a
>>valid upgrade image loaded into the unit.
>>
>> 1. Erase the application image from the second image bank (still can
>>recover since the upgrade image is valid)
>> 2. Upload a new upgrade image to the second image bank (primary
>>upgrade image still intact in case secondary fails)
>> 3. Swap the upgrade images in the 1st and 2nd bank (if secondary
>>doesn’t boot, we can always revert)
>> 4. Load the new application into the 2nd bank (if this fails, we
>>still have the upgraded in the primary)
>> 5. Complete
>
>--
>David G. Simmons
>(919) 534-5099
>Web • Blog • Linkedin • Twitter • GitHub
>/** Message digitally signed for security and authenticity.
>* If you cannot read the PGP.sig attachment, please go to
> * http://www.gnupg.com/ Secure your email!!!
> * Public key available at keyserver.pgp.com
>**/
>♺ This email uses 100% recycled electrons. Don't blow it by printing!
>
>There are only 2 hard things in computer science: Cache invalidation,
>naming things, and off-by-one errors.
>
>
Re: split image to increase application storage space.
Posted by "David G. Simmons" <sa...@mac.com>.
+1 This is very similar to how we handled upgrading images on a previous IoT platform and it was very good at keeping customers from bricking their device. I also like that it allows for the upgrading of just the ADIC without touching a valid and functioning BLE Stack, etc. which makes it more robust in that there is a lower chance of intruducing a lower-level bug into the system simply by updating the application.
dg
> On Jun 21, 2016, at 3:31 PM, paul@wrada.com wrote:
>
> I’m working on split image feature and I think I just have one more major design issue to consider, and that is newt build related.
>
> First, a summary for folks who are unaware of this effort. The goal is to create an application in two pieces to fit into two image banks such that one piece would contain the bluetooth stack and firmware update application and the other would contain the primary customer application (that’s the goal, but it would be defined generally to allow any split). These two are linked together with a special property that the upgrade app could run without the primary customer app but not vice versa. To give these names I call the independent one the AIIC (Application independent image component) and the customer app the ADIC (Application Dependent Image component). This would allow the following upgrade procedure with two flash images. At each step, there would always be a valid upgrade image loaded into the unit.
>
> 1. Erase the application image from the second image bank (still can recover since the upgrade image is valid)
> 2. Upload a new upgrade image to the second image bank (primary upgrade image still intact in case secondary fails)
> 3. Swap the upgrade images in the 1st and 2nd bank (if secondary doesn’t boot, we can always revert)
> 4. Load the new application into the 2nd bank (if this fails, we still have the upgraded in the primary)
> 5. Complete
--
David G. Simmons
(919) 534-5099
Web • Blog • Linkedin • Twitter • GitHub
/** Message digitally signed for security and authenticity.
* If you cannot read the PGP.sig attachment, please go to
* http://www.gnupg.com/ Secure your email!!!
* Public key available at keyserver.pgp.com
**/
♺ This email uses 100% recycled electrons. Don't blow it by printing!
There are only 2 hard things in computer science: Cache invalidation, naming things, and off-by-one errors.
Re: split image to increase application storage space.
Posted by will sanfilippo <wi...@runtime.io>.
Yep; that clarifies things for me. Thanks!
> On Jun 21, 2016, at 2:09 PM, paul@wrada.com wrote:
>
>
> The example was simplified to give folks the general idea.
>
> Practically, the loader=AIIC code would include whatever someone wanted it
> to include and it would compile and link as a stand alone app; in this
> case I used Bluetooth stack and firmware upgrade as an example. The AIDC
> would contain the stuff that is NOT in the AIIC.
>
> So to give a more complex case, suppose the AIIC used 60% of the bluetooth
> stack, that¹s all that would be included in there. If the application
> used 100% of the bluetooth stack, the remainder would be in the AIDC.
>
> This is a consequence of linking the AIIC as a stand-alone app, and then
> linking the AIDC using all the symbols present in the AIIC.
>
> Does that help or make it more confusing?
>
> Code would not be upgraded independently, as the AIDC is hard-linked to
> fixed symbol addresses in the AIIC. While it might be possible to create
> a new AIDC that used the same symbols from the AIIC, I was not intending
> to allow that.
>
> Paul
>
> On 6/21/16, 1:54 PM, "will sanfilippo" <wi...@runtime.io> wrote:
>
>> +1 on idea 1.
>>
>> Just a point of clarification though: why do you specfically break out
>> (in your example) the bluetooth stack size and the upgrade code size? I
>> think I was originally thinking about things slightly differently here in
>> that the bluetooth stack and upgrade image were combined into one ³image²
>> (they would not be separate) but that does not appear to be the case
>> here. Well, not sure actually. Can you upgrade the upgrade code
>> independently? Sorry if this is obvious and I am not quite getting it :-)
>>
>>> On Jun 21, 2016, at 12:31 PM, paul@wrada.com wrote:
>>>
>>> I¹m working on split image feature and I think I just have one more
>>> major design issue to consider, and that is newt build related.
>>>
>>> First, a summary for folks who are unaware of this effort. The goal is
>>> to create an application in two pieces to fit into two image banks such
>>> that one piece would contain the bluetooth stack and firmware update
>>> application and the other would contain the primary customer application
>>> (that¹s the goal, but it would be defined generally to allow any split).
>>> These two are linked together with a special property that the upgrade
>>> app could run without the primary customer app but not vice versa. To
>>> give these names I call the independent one the AIIC (Application
>>> independent image component) and the customer app the ADIC (Application
>>> Dependent Image component). This would allow the following upgrade
>>> procedure with two flash images. At each step, there would always be a
>>> valid upgrade image loaded into the unit.
>>>
>>> 1. Erase the application image from the second image bank (still can
>>> recover since the upgrade image is valid)
>>> 2. Upload a new upgrade image to the second image bank (primary
>>> upgrade image still intact in case secondary fails)
>>> 3. Swap the upgrade images in the 1st and 2nd bank (if secondary
>>> doesn¹t boot, we can always revert)
>>> 4. Load the new application into the 2nd bank (if this fails, we
>>> still have the upgraded in the primary)
>>> 5. Complete
>>>
>>> This model allows a safe upgrade (because the upgrade image can always
>>> upgrade) while preserving more space for the application because the
>>> application doesn¹t have to duplicate the bluetooth stack and associated
>>> upgrade code.
>>>
>>> Consider this example, two 112k flash banks which we want to store a
>>> bluetooth application. Assume the following code sizes: Application size
>>> 32k, Bluetooth stack size 64k, Upgrade code size 16k. With two
>>> independent app images, each would be 112k filling the available
>>> sectors. However, if we split the image we would have an upgrade image
>>> of 80k and an app image of 32k (since it uses bluetooth and upgrade
>>> from the AIIC) leaving tons more space for sophisticated applications
>>> and more space for the upgrade as well.
>>>
>>> We decided to create this split image as a pair that are linked
>>> together during the build process. There will be no dynamic bindings
>>> like SWI or function table or anything like that. The goal is not to
>>> separate the OS/stack from the app, but just to be a bit more efficient
>>> about duplicating the large components like bluetooth. Thus, when up are
>>> updating one of the split images, you MUST update the other. Splitting
>>> the image makes this ³safe" as there will always be a bank filled with
>>> an image that is capable of performing an upgrade.
>>>
>>> Anyway, that was the background, now on to my design question
>>>
>>> The question I am investigating is how to formulate this in the newt
>>> build process. Here are two alternatives.
>>>
>>> IDEA1: I was thinking that I would have a single target to produce both
>>> of these image components. The target would still have one bsp, one
>>> build profile, but I would define a new token for the target called
>>> loader=xxx. This would define the loader AIIC that would go into the
>>> first image bank, and if ³loader" was defined, the app would be linked
>>> to run in the second image bank. Thus, if I were building a bluetooth
>>> app called foo, I would set loader=AIIC (contains upgrade and
>>> bluetooth) and then set app=foo. If ³loader² was not defined, I would
>>> ensure the newt tool would do a complete build of foo and link it into
>>> the first image bank.
>>>
>>> IDEA2: The alternative would be to have two separate targets. The AIDC
>>> target would have to reference the AIIC target and know that it was
>>> being built to be dependent on that bit. In this case, the AIIC target
>>> would be build as normal with no changes to the build system. Then the
>>> AIDC target would have to have a new tag that said that it was dependent
>>> on the other target. For example, in the app:foo target descriptor it
>>> would say split_image_dep=AIIC. This would cause newt to perform some
>>> special processing of the AIIC elf image, and ensure that we link the
>>> foo AIDC app to run out of the second image bank and reference common
>>> functions in the first image bank.
>>>
>>> In either case, the application will be build as a split image and will
>>> contain two elf files. One file will be the Application Independent
>>> AIIC) part, and another will be the application dependent (ADIC) part.
>>> These will both get converted to image files by newt create-image.
>>>
>>> Comments? My thoughts are that IDEA1 is the better of these two. It
>>> would allow newt to be split-image-aware during create-image and during
>>> download and debug. Any other ideas from someone more knowledgeable
>>> about newt?
>>>
>>> Paul
>>>
>>>
>>
>
Re: split image to increase application storage space.
Posted by "paul@wrada.com" <pa...@wrada.com>.
The example was simplified to give folks the general idea.
Practically, the loader=AIIC code would include whatever someone wanted it
to include and it would compile and link as a stand alone app; in this
case I used Bluetooth stack and firmware upgrade as an example. The AIDC
would contain the stuff that is NOT in the AIIC.
So to give a more complex case, suppose the AIIC used 60% of the bluetooth
stack, that¹s all that would be included in there. If the application
used 100% of the bluetooth stack, the remainder would be in the AIDC.
This is a consequence of linking the AIIC as a stand-alone app, and then
linking the AIDC using all the symbols present in the AIIC.
Does that help or make it more confusing?
Code would not be upgraded independently, as the AIDC is hard-linked to
fixed symbol addresses in the AIIC. While it might be possible to create
a new AIDC that used the same symbols from the AIIC, I was not intending
to allow that.
Paul
On 6/21/16, 1:54 PM, "will sanfilippo" <wi...@runtime.io> wrote:
>+1 on idea 1.
>
>Just a point of clarification though: why do you specfically break out
>(in your example) the bluetooth stack size and the upgrade code size? I
>think I was originally thinking about things slightly differently here in
>that the bluetooth stack and upgrade image were combined into one ³image²
>(they would not be separate) but that does not appear to be the case
>here. Well, not sure actually. Can you upgrade the upgrade code
>independently? Sorry if this is obvious and I am not quite getting it :-)
>
>> On Jun 21, 2016, at 12:31 PM, paul@wrada.com wrote:
>>
>> I¹m working on split image feature and I think I just have one more
>>major design issue to consider, and that is newt build related.
>>
>> First, a summary for folks who are unaware of this effort. The goal is
>>to create an application in two pieces to fit into two image banks such
>>that one piece would contain the bluetooth stack and firmware update
>>application and the other would contain the primary customer application
>>(that¹s the goal, but it would be defined generally to allow any split).
>> These two are linked together with a special property that the upgrade
>>app could run without the primary customer app but not vice versa. To
>>give these names I call the independent one the AIIC (Application
>>independent image component) and the customer app the ADIC (Application
>>Dependent Image component). This would allow the following upgrade
>>procedure with two flash images. At each step, there would always be a
>>valid upgrade image loaded into the unit.
>>
>> 1. Erase the application image from the second image bank (still can
>>recover since the upgrade image is valid)
>> 2. Upload a new upgrade image to the second image bank (primary
>>upgrade image still intact in case secondary fails)
>> 3. Swap the upgrade images in the 1st and 2nd bank (if secondary
>>doesn¹t boot, we can always revert)
>> 4. Load the new application into the 2nd bank (if this fails, we
>>still have the upgraded in the primary)
>> 5. Complete
>>
>> This model allows a safe upgrade (because the upgrade image can always
>>upgrade) while preserving more space for the application because the
>>application doesn¹t have to duplicate the bluetooth stack and associated
>>upgrade code.
>>
>> Consider this example, two 112k flash banks which we want to store a
>>bluetooth application. Assume the following code sizes: Application size
>>32k, Bluetooth stack size 64k, Upgrade code size 16k. With two
>>independent app images, each would be 112k filling the available
>>sectors. However, if we split the image we would have an upgrade image
>>of 80k and an app image of 32k (since it uses bluetooth and upgrade
>>from the AIIC) leaving tons more space for sophisticated applications
>>and more space for the upgrade as well.
>>
>> We decided to create this split image as a pair that are linked
>>together during the build process. There will be no dynamic bindings
>>like SWI or function table or anything like that. The goal is not to
>>separate the OS/stack from the app, but just to be a bit more efficient
>>about duplicating the large components like bluetooth. Thus, when up are
>>updating one of the split images, you MUST update the other. Splitting
>>the image makes this ³safe" as there will always be a bank filled with
>>an image that is capable of performing an upgrade.
>>
>> Anyway, that was the background, now on to my design question
>>
>> The question I am investigating is how to formulate this in the newt
>>build process. Here are two alternatives.
>>
>> IDEA1: I was thinking that I would have a single target to produce both
>>of these image components. The target would still have one bsp, one
>>build profile, but I would define a new token for the target called
>>loader=xxx. This would define the loader AIIC that would go into the
>>first image bank, and if ³loader" was defined, the app would be linked
>>to run in the second image bank. Thus, if I were building a bluetooth
>>app called foo, I would set loader=AIIC (contains upgrade and
>>bluetooth) and then set app=foo. If ³loader² was not defined, I would
>>ensure the newt tool would do a complete build of foo and link it into
>>the first image bank.
>>
>> IDEA2: The alternative would be to have two separate targets. The AIDC
>>target would have to reference the AIIC target and know that it was
>>being built to be dependent on that bit. In this case, the AIIC target
>>would be build as normal with no changes to the build system. Then the
>>AIDC target would have to have a new tag that said that it was dependent
>>on the other target. For example, in the app:foo target descriptor it
>>would say split_image_dep=AIIC. This would cause newt to perform some
>>special processing of the AIIC elf image, and ensure that we link the
>>foo AIDC app to run out of the second image bank and reference common
>>functions in the first image bank.
>>
>> In either case, the application will be build as a split image and will
>>contain two elf files. One file will be the Application Independent
>>AIIC) part, and another will be the application dependent (ADIC) part.
>>These will both get converted to image files by newt create-image.
>>
>> Comments? My thoughts are that IDEA1 is the better of these two. It
>>would allow newt to be split-image-aware during create-image and during
>>download and debug. Any other ideas from someone more knowledgeable
>>about newt?
>>
>> Paul
>>
>>
>
Re: split image to increase application storage space.
Posted by will sanfilippo <wi...@runtime.io>.
+1 on idea 1.
Just a point of clarification though: why do you specfically break out (in your example) the bluetooth stack size and the upgrade code size? I think I was originally thinking about things slightly differently here in that the bluetooth stack and upgrade image were combined into one “image” (they would not be separate) but that does not appear to be the case here. Well, not sure actually. Can you upgrade the upgrade code independently? Sorry if this is obvious and I am not quite getting it :-)
> On Jun 21, 2016, at 12:31 PM, paul@wrada.com wrote:
>
> I’m working on split image feature and I think I just have one more major design issue to consider, and that is newt build related.
>
> First, a summary for folks who are unaware of this effort. The goal is to create an application in two pieces to fit into two image banks such that one piece would contain the bluetooth stack and firmware update application and the other would contain the primary customer application (that’s the goal, but it would be defined generally to allow any split). These two are linked together with a special property that the upgrade app could run without the primary customer app but not vice versa. To give these names I call the independent one the AIIC (Application independent image component) and the customer app the ADIC (Application Dependent Image component). This would allow the following upgrade procedure with two flash images. At each step, there would always be a valid upgrade image loaded into the unit.
>
> 1. Erase the application image from the second image bank (still can recover since the upgrade image is valid)
> 2. Upload a new upgrade image to the second image bank (primary upgrade image still intact in case secondary fails)
> 3. Swap the upgrade images in the 1st and 2nd bank (if secondary doesn’t boot, we can always revert)
> 4. Load the new application into the 2nd bank (if this fails, we still have the upgraded in the primary)
> 5. Complete
>
> This model allows a safe upgrade (because the upgrade image can always upgrade) while preserving more space for the application because the application doesn’t have to duplicate the bluetooth stack and associated upgrade code.
>
> Consider this example, two 112k flash banks which we want to store a bluetooth application. Assume the following code sizes: Application size 32k, Bluetooth stack size 64k, Upgrade code size 16k. With two independent app images, each would be 112k filling the available sectors. However, if we split the image we would have an upgrade image of 80k and an app image of 32k (since it uses bluetooth and upgrade from the AIIC) leaving tons more space for sophisticated applications and more space for the upgrade as well.
>
> We decided to create this split image as a pair that are linked together during the build process. There will be no dynamic bindings like SWI or function table or anything like that. The goal is not to separate the OS/stack from the app, but just to be a bit more efficient about duplicating the large components like bluetooth. Thus, when up are updating one of the split images, you MUST update the other. Splitting the image makes this “safe" as there will always be a bank filled with an image that is capable of performing an upgrade.
>
> Anyway, that was the background, now on to my design question
>
> The question I am investigating is how to formulate this in the newt build process. Here are two alternatives.
>
> IDEA1: I was thinking that I would have a single target to produce both of these image components. The target would still have one bsp, one build profile, but I would define a new token for the target called loader=xxx. This would define the loader AIIC that would go into the first image bank, and if “loader" was defined, the app would be linked to run in the second image bank. Thus, if I were building a bluetooth app called foo, I would set loader=AIIC (contains upgrade and bluetooth) and then set app=foo. If “loader” was not defined, I would ensure the newt tool would do a complete build of foo and link it into the first image bank.
>
> IDEA2: The alternative would be to have two separate targets. The AIDC target would have to reference the AIIC target and know that it was being built to be dependent on that bit. In this case, the AIIC target would be build as normal with no changes to the build system. Then the AIDC target would have to have a new tag that said that it was dependent on the other target. For example, in the app:foo target descriptor it would say split_image_dep=AIIC. This would cause newt to perform some special processing of the AIIC elf image, and ensure that we link the foo AIDC app to run out of the second image bank and reference common functions in the first image bank.
>
> In either case, the application will be build as a split image and will contain two elf files. One file will be the Application Independent AIIC) part, and another will be the application dependent (ADIC) part. These will both get converted to image files by newt create-image.
>
> Comments? My thoughts are that IDEA1 is the better of these two. It would allow newt to be split-image-aware during create-image and during download and debug. Any other ideas from someone more knowledgeable about newt?
>
> Paul
>
>