Re: FCB revamp

[Laczen JMS <la...@gmail.com>]

Hi Sterling,

Op di 31 jul. 2018 om 23:11 schreef Sterling Hughes
<st...@gmail.com>:
>
> Hi,
>
> On 31 Jul 2018, at 0:47, Laczen JMS wrote:
>
> > Hi Marko and Will,
> >
> > I have experienced the same problem while developing nvs for zephyr. I
> > always got stuck when I would except that there would be changes to
> > flash by an "external" factor. As soon as this could happen there is a
> > problem with the meaning of the crc. The crc fails this means:
> > a. Incomplete write,
> > b. Some tampering occurred,
> > c. The len is not pointing to the correct location,
> > In all these cases the result will be that the data is not considered
> > valid, the only case where a increased crc might be helping is in b.
>
> I think (b) is more commonly failure of flash over time, and writes
> and/or erases that fail.  Some times bits will be “stuck” on
> flashes, especially inexpensive ones in high volume consumer products
> that have no correction for this.  The probability that this will happen
> across a population of millions of devices is probably pretty high, and
> the bugs are hard to find.
>
The failure were bits will be stuck is both (b) and (c). There is no reason
why the stuck bits would not appear in the len data.
> >
> > In the latest version of nvs I have changed how the data is written to
> > flash. The data itself is written from the start of a sector, a
> > "allocation entry" is written from the end of a flash sector. This
> > allocation entry contains a sector offset and length of the data
> > written.  The allocation entry is of fixed size. The writing of data
> > is done using the following method:
> > 1. Check if there is sufficient space (difference between allocation
> > entry location and next data location),
> > 2. Write the data at the specific location, this could include a data
> > crc,
> > 3. Write the allocation entry, this includes a crc calculated over the
> > item length and offset,
> >
> > When there is insufficient space a allocation entry of all zeros is
> > written at the end of the allocation entries, and a new sector is
> > used.
> >
> > At startup the allocation entry write position is found by stepping
> > through the allocation entries until all 0xff are found. The data
> > write position is found by searching the position from which all data
> > up to the allocation entry write position is 0xff.
> >
>
> This is a good way of accomplishing this, another is to use a magic on
> every entry.  It would be interesting to allow streaming of entries
> using this approach — fixed length entries that contain type, offset,
> len — perhaps encoded more efficiently.  We’ve been discussing
> adding a flash store interface to Mynewt similar to NVS and one of the
> requirements is definitely to not require the full amount of RAM to
> write a larger BLOB entry to the flash.
I guess by magic you mean some kind of recognizable start record, this
would certainly limit the data you can store in fcb.

Maybe we can team up to create a fcb2, taking some ideas from fcb and
some of nvs and create a basis that is easily extended to different kind of
records to be stored in flash.

I am thinking about:
a. using a allocation entry table at the end of a sector
b. define the allocation entry as:
    type + len + offset + ate crc (1 byte) + data crc (2 bytes)
    where: type can be used to determine what is inside the data:
        0: record without specific meaning (fcb today)
        1: record contains data id (16 bit) followed by data
        2: record contains data name (string finished by \0) followed by data
c. nvs and possibly other storage solutions are added on top of fcb2.
d. write in such a way that streaming is allowed

Kind regards,

Jehudi
>
> Sterling

Re: FCB revamp

[Sterling Hughes <st...@gmail.com>]

Hi,

>>>
>>
>> I think (b) is more commonly failure of flash over time, and writes
>> and/or erases that fail.  Some times bits will be “stuck” on
>> flashes, especially inexpensive ones in high volume consumer products
>> that have no correction for this.  The probability that this will 
>> happen
>> across a population of millions of devices is probably pretty high, 
>> and
>> the bugs are hard to find.
>>
> The failure were bits will be stuck is both (b) and (c). There is no 
> reason
> why the stuck bits would not appear in the len data.

Yup, my concern is making sure that invalid data is not passed up to the 
higher layer, which I think would be the case with FCB now.  I think in 
production, over a population of devices, this will manifest itself as 
random crashes.

>>
> I guess by magic you mean some kind of recognizable start record, this
> would certainly limit the data you can store in fcb.
>
> Maybe we can team up to create a fcb2, taking some ideas from fcb and
> some of nvs and create a basis that is easily extended to different 
> kind of
> records to be stored in flash.
>

Definitely, I think this would be great.

> I am thinking about:
> a. using a allocation entry table at the end of a sector
> b. define the allocation entry as:
>     type + len + offset + ate crc (1 byte) + data crc (2 bytes)
>     where: type can be used to determine what is inside the data:
>         0: record without specific meaning (fcb today)
>         1: record contains data id (16 bit) followed by data
>         2: record contains data name (string finished by \0) followed 
> by data
> c. nvs and possibly other storage solutions are added on top of fcb2.
> d. write in such a way that streaming is allowed
>

I agree.  I think interleaving the various types is also important (as 
you reference above), as I think it would be good to share flash storage 
areas with one big FCB that could be used for config/other elements.

Sterling