Trouble with config + nffs

[Simon Ratner <si...@proxy.co>]

Hi list,

I am trying to use config/config.h and config/config_file.h, based on the
slinky example, to store some persistent data in flash fs on nrf51.

Some of the values are written once and remain static after that (e.g.
device id), while other values might change over the lifetime of the device
(e.g. actual configurable settings that need to persist across power
cycles).

The problem I am having is as follows:

1. Write initial values at provisioning time via serial console:

config x/a 1
config x/b 2
config x/c 3
config commit
config save

These values are saved and I can verify that /cfg/run has the right bits.

2. At some later point, change x/b. I have tried doing this both via code
(calling config_save) and via the console, with same result:

console x/b 10
console commit
console save

At this point, only the new value is saved, and /cfg/run no longer contains
any of the previous values, even though config dump still has them all (ie.
the values in memory are correct, and export handler returns them). Of
course, after a power cycle, I lose my previously persistent settings.

Worse, if at this point I try to manually re-set all settings to their
values, all except x/b are saved, so it seems the values are now in two
disjoint sets, and only one of the sets is saved on any particular call to
config_save.

Any ideas on what is going, or pointers to what I am doing wrong? Is this a
side effect of the way flash blocks are written/erased?

PS. As a side note, I have observed some other general flakiness with
config + nffs. Sometimes the initial write partially fails and only some of
the data is saved, seems to be related to the file straddling the inode
boundary (only the data in the second inode can be read back). At other
times, nffs_detect fails after a previous write. Might all be related?

Cheers,
Simon.

Re: Trouble with config + nffs

[Simon Ratner <si...@proxy.co>]

> The issue was that config save was blowing away all of the previous stuff.
> I.e. I imagined that all modules would every time ‘save’ gets called would
> store the configuration they want to keep around.
>
> But after using it like that for a bit decided that this is not a good
> model.
> And instead ‘save’ will keep appending to existing configuration. Or
> replace the previous value.
>

Actually, even if you blow away the previous file, couldn't you re-write
all the values as currently returned by export, rather than just the ones
you set in this "session"? The data is still there until reboot.

Re: Trouble with config + nffs

[marko kiiskila <ma...@runtime.io>]

That is what the ‘commit’ call is supposed to be used for.

E.g. at the end of the call to conf_load() (which can
load from multiple files, btw) it calls ‘commit’ for all the modules
which have registered handlers. In theory you should be able
to use this as the indicator that it’s time to take the input config,
and act on it.

I’m still ambivalent on whether the conf_load() should call
the config handlers only once, or if it’s ok to call them multiple
times. Upside is that there would be only one call, downside
is that we’ll end up walking the flash more. Which can slow
down the bootup process quite a bit if flash is behind SPI.

I could cache the names and locations during the file/FCB walk,
and then read in only those values, but that needs temporary
RAM space, which is very precious.

> On Jun 9, 2016, at 5:50 PM, Simon Ratner <si...@proxy.co> wrote:
> 
> Thanks! I'll check out develop.
> 
> Regarding multiple load events, what do you think of the equivalent of
> conf_commit that is called for the value after you know that you have
> loaded the latest copy? This isn't currently a concern for me, so just an
> idea for other use cases which might have dependencies between config
> values that must be applied together.
> On Jun 9, 2016 5:20 PM, "marko kiiskila" <ma...@runtime.io> wrote:
> 
> Hi Simon,
> 
>> On Jun 9, 2016, at 4:59 PM, Simon Ratner <si...@proxy.co> wrote:
>> 
> ...
>> 
>> At this point, only the new value is saved, and /cfg/run no longer
> contains
>> any of the previous values, even though config dump still has them all
> (ie.
>> the values in memory are correct, and export handler returns them). Of
>> course, after a power cycle, I lose my previously persistent settings.
>> 
>> Worse, if at this point I try to manually re-set all settings to their
>> values, all except x/b are saved, so it seems the values are now in two
>> disjoint sets, and only one of the sets is saved on any particular call to
>> config_save.
>> 
>> Any ideas on what is going, or pointers to what I am doing wrong? Is this
> a
>> side effect of the way flash blocks are written/erased?
> 
> The issue was that config save was blowing away all of the previous stuff.
> I.e. I imagined that all modules would every time ‘save’ gets called would
> store the configuration they want to keep around.
> 
> But after using it like that for a bit decided that this is not a good
> model.
> And instead ‘save’ will keep appending to existing configuration. Or
> replace the previous value.
> 
> I just pushed changes to develop which should make it behave like you
> expect.
> 
> There’s still one wart that I’m aware of; during config load you’ll get
> called
> multiple times, if you’ve stored the config item with different value. The
> last
> call is the one that has the latest saved value.
> 
>> PS. As a side note, I have observed some other general flakiness with
>> config + nffs. Sometimes the initial write partially fails and only some
> of
>> the data is saved, seems to be related to the file straddling the inode
>> boundary (only the data in the second inode can be read back). At other
>> times, nffs_detect fails after a previous write. Might all be related?
> 
> That sounds like a bug in NFFS, probably worth pursuing. I don’t think
> that’s related to config behavior.

Re: Trouble with config + nffs

[Simon Ratner <si...@proxy.co>]

Thanks! I'll check out develop.

Regarding multiple load events, what do you think of the equivalent of
conf_commit that is called for the value after you know that you have
loaded the latest copy? This isn't currently a concern for me, so just an
idea for other use cases which might have dependencies between config
values that must be applied together.
On Jun 9, 2016 5:20 PM, "marko kiiskila" <ma...@runtime.io> wrote:

Hi Simon,

> On Jun 9, 2016, at 4:59 PM, Simon Ratner <si...@proxy.co> wrote:
>
...
>
> At this point, only the new value is saved, and /cfg/run no longer
contains
> any of the previous values, even though config dump still has them all
(ie.
> the values in memory are correct, and export handler returns them). Of
> course, after a power cycle, I lose my previously persistent settings.
>
> Worse, if at this point I try to manually re-set all settings to their
> values, all except x/b are saved, so it seems the values are now in two
> disjoint sets, and only one of the sets is saved on any particular call to
> config_save.
>
> Any ideas on what is going, or pointers to what I am doing wrong? Is this
a
> side effect of the way flash blocks are written/erased?

The issue was that config save was blowing away all of the previous stuff.
I.e. I imagined that all modules would every time ‘save’ gets called would
store the configuration they want to keep around.

But after using it like that for a bit decided that this is not a good
model.
And instead ‘save’ will keep appending to existing configuration. Or
replace the previous value.

I just pushed changes to develop which should make it behave like you
expect.

There’s still one wart that I’m aware of; during config load you’ll get
called
multiple times, if you’ve stored the config item with different value. The
last
call is the one that has the latest saved value.

> PS. As a side note, I have observed some other general flakiness with
> config + nffs. Sometimes the initial write partially fails and only some
of
> the data is saved, seems to be related to the file straddling the inode
> boundary (only the data in the second inode can be read back). At other
> times, nffs_detect fails after a previous write. Might all be related?

That sounds like a bug in NFFS, probably worth pursuing. I don’t think
that’s related to config behavior.

Re: Trouble with config + nffs

[marko kiiskila <ma...@runtime.io>]

Hi Simon,

> On Jun 9, 2016, at 4:59 PM, Simon Ratner <si...@proxy.co> wrote:
> 
...
> 
> At this point, only the new value is saved, and /cfg/run no longer contains
> any of the previous values, even though config dump still has them all (ie.
> the values in memory are correct, and export handler returns them). Of
> course, after a power cycle, I lose my previously persistent settings.
> 
> Worse, if at this point I try to manually re-set all settings to their
> values, all except x/b are saved, so it seems the values are now in two
> disjoint sets, and only one of the sets is saved on any particular call to
> config_save.
> 
> Any ideas on what is going, or pointers to what I am doing wrong? Is this a
> side effect of the way flash blocks are written/erased?

The issue was that config save was blowing away all of the previous stuff.
I.e. I imagined that all modules would every time ‘save’ gets called would
store the configuration they want to keep around.

But after using it like that for a bit decided that this is not a good model.
And instead ‘save’ will keep appending to existing configuration. Or
replace the previous value.

I just pushed changes to develop which should make it behave like you
expect.

There’s still one wart that I’m aware of; during config load you’ll get called
multiple times, if you’ve stored the config item with different value. The last
call is the one that has the latest saved value.

> PS. As a side note, I have observed some other general flakiness with
> config + nffs. Sometimes the initial write partially fails and only some of
> the data is saved, seems to be related to the file straddling the inode
> boundary (only the data in the second inode can be read back). At other
> times, nffs_detect fails after a previous write. Might all be related?

That sounds like a bug in NFFS, probably worth pursuing. I don’t think
that’s related to config behavior.