You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nuttx.apache.org by Saurav Pal <re...@gmail.com> on 2024/09/12 15:48:17 UTC
GSoC Final Report on mnemofs
Hi all,
Here's my final report <https://resyfer.github.io/blogs/mnemofs/endeval/>
on mnemofs, a NAND flash file system for NuttX, on which I worked during my
tenure as a GSoC 2024 Contributor for ASF. I would be grateful for any
suggestions and criticism.
Best regards,
Saurav Pal.
Re: GSoC Final Report on mnemofs
Posted by "Alan C. Assis" <ac...@gmail.com>.
Hi Saurav,
Congratulations for your great work!
Everyone, please take a look at Saurav report and if you can: run the
mnemofs SIM NAND example.
Now we need to test it in a real device (we don't have any NAND driver
yet), maybe we can start using MnemoFS with SPI NOR Flash.
Any help is appreciated!
BR,
Alan
On Thu, Sep 12, 2024 at 12:49 PM Saurav Pal <re...@gmail.com> wrote:
> Hi all,
>
> Here's my final report <https://resyfer.github.io/blogs/mnemofs/endeval/>
> on mnemofs, a NAND flash file system for NuttX, on which I worked during my
> tenure as a GSoC 2024 Contributor for ASF. I would be grateful for any
> suggestions and criticism.
>
> Best regards,
> Saurav Pal.
>
Re: GSoC Final Report on mnemofs
Posted by Tomek CEDRO <to...@cedro.info>.
On Thu, Sep 12, 2024 at 5:49 PM Saurav Pal <re...@gmail.com> wrote:
> Hi all,
> Here's my final report <https://resyfer.github.io/blogs/mnemofs/endeval/>
> on mnemofs, a NAND flash file system for NuttX, on which I worked during my
> tenure as a GSoC 2024 Contributor for ASF. I would be grateful for any
> suggestions and criticism.
> Best regards,
> Saurav Pal.
Congratulations and Thank You Saurav :-) We hope to see more commits
for NuttX very soon :-)
--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
Re: GSoC Final Report on mnemofs
Posted by "Alan C. Assis" <ac...@gmail.com>.
AFAIK the AI bot is used only to help identify things that are missing in
the PR, so it avoids asking users again and again to fill the Summary, etc.
BTW, I don't think simple things like that need Apache or PMC approval.
It is an open-source project after all and contributions are welcome!
But I'm glad we have you to disagree, it means we are doing the right thing
("Unanimity is always stupid." -- Nelson Rodrigues).
BR,
Alan
On Thu, Oct 3, 2024 at 5:50 AM Sebastien Lorquet <se...@lorquet.fr>
wrote:
> Hi,
>
> I am now the owner of 4 Micron NAND chips and I am at the disposal of
> NuttX developers who want to test code to drive and use them.
>
>
> Tomek: It doesnt change a damn thing.
>
> If LLM are used to make up for contributor laziness that's even worse.
> You'll get unreliable slop when the proper thing to do is close these
> bogus requests and ask contributors to fill actual required information.
>
> Was the use of ai discussed and voted by the project? No need, I'll be
> the only one against it?
>
> Sebastien
>
>
> On 01/10/2024 13:01, Tomek CEDRO wrote:
> > On Tue, Oct 1, 2024 at 10:28 AM Sebastien Lorquet <se...@lorquet.fr>
> wrote:
> >> (..)
> >> I will not be able to write new code. Specifically if it's now AI
> reviewed.
> > Sebastien the code is not reviewed by the AI it only provides initial
> > feedback on the PR and commit descriptions something we are constantly
> > repeating and asking some reporters to fill in with meaningful content
> > and what should be done correctly in the first place but its often not
> > :-) Humans are still reviewing, commenting, requesting changes,
> > approving, and merging the code no worries :-)
> >
>
Re: GSoC Final Report on mnemofs
Posted by Sebastien Lorquet <se...@lorquet.fr>.
Hi,
I am now the owner of 4 Micron NAND chips and I am at the disposal of
NuttX developers who want to test code to drive and use them.
Tomek: It doesnt change a damn thing.
If LLM are used to make up for contributor laziness that's even worse.
You'll get unreliable slop when the proper thing to do is close these
bogus requests and ask contributors to fill actual required information.
Was the use of ai discussed and voted by the project? No need, I'll be
the only one against it?
Sebastien
On 01/10/2024 13:01, Tomek CEDRO wrote:
> On Tue, Oct 1, 2024 at 10:28 AM Sebastien Lorquet <se...@lorquet.fr> wrote:
>> (..)
>> I will not be able to write new code. Specifically if it's now AI reviewed.
> Sebastien the code is not reviewed by the AI it only provides initial
> feedback on the PR and commit descriptions something we are constantly
> repeating and asking some reporters to fill in with meaningful content
> and what should be done correctly in the first place but its often not
> :-) Humans are still reviewing, commenting, requesting changes,
> approving, and merging the code no worries :-)
>
Re: GSoC Final Report on mnemofs
Posted by Tomek CEDRO <to...@cedro.info>.
On Tue, Oct 1, 2024 at 10:28 AM Sebastien Lorquet <se...@lorquet.fr> wrote:
> (..)
> I will not be able to write new code. Specifically if it's now AI reviewed.
Sebastien the code is not reviewed by the AI it only provides initial
feedback on the PR and commit descriptions something we are constantly
repeating and asking some reporters to fill in with meaningful content
and what should be done correctly in the first place but its often not
:-) Humans are still reviewing, commenting, requesting changes,
approving, and merging the code no worries :-)
--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
Re: GSoC Final Report on mnemofs
Posted by Sebastien Lorquet <se...@lorquet.fr>.
Hello,
The following NAND flash chips have been ordered today:
MT29F2G01ABAGDWB-IT:G
MT29F1G01ABAFDWB-IT:F
MT29F4G01ABAFDWB-IT:F
I have selected these references because, they are biiiig and
potentially useful, and the 2Gbit chip seems to have a different memory
layout.
I will not be able to write new code. Specifically if it's now AI reviewed.
But I will be able to run test code of your choosing on a raspi pico or
nucleo-l432kc
Sebastien
On 13/09/2024 22:43, Sebastien Lorquet wrote:
> Hello,
>
> Thank you for the additional information, of course I understand the
> gsoc had to be completed in a limited time, and the current code is a
> prototype.
>
> I think it will be cool if we manage to make it production quality for
> all users of the nuttx codebase.
>
> I will probably send my farnell order next week so I'll have some
> hardware to test with. the 4gbit micron chip is much larger than a nor
> flash, and using it with a proper resilient fs will be nice.
>
> If I get help with required software, we'll make it work.
>
>
> On 9/13/24 22:09, Saurav Pal wrote:
>> Assuming the bytes are written to the flash from the start to
>> end (which I have not seen anything to suggest otherwise)
>
> <terminator voice>Theoretically.</>
>
> Joke apart, yes, it's a reasonable assumption. Sometimes it's word per
> word not byte per byte, and even if the first bytes written could be
> at the last address, I have never seen it done. However it is a good
> design practice to explicitly separate the writing of the final CRC,
> to make it obvious that you expect the previous write to be complete.
>
> The design you described should work and inspires confidence.
>
> Sebastien
>
>
Re: GSoC Final Report on mnemofs
Posted by Sebastien Lorquet <se...@lorquet.fr>.
Hello,
Thank you for the additional information, of course I understand the
gsoc had to be completed in a limited time, and the current code is a
prototype.
I think it will be cool if we manage to make it production quality for
all users of the nuttx codebase.
I will probably send my farnell order next week so I'll have some
hardware to test with. the 4gbit micron chip is much larger than a nor
flash, and using it with a proper resilient fs will be nice.
If I get help with required software, we'll make it work.
On 9/13/24 22:09, Saurav Pal wrote:
> Assuming the bytes are written to the flash from the start to
> end (which I have not seen anything to suggest otherwise)
<terminator voice>Theoretically.</>
Joke apart, yes, it's a reasonable assumption. Sometimes it's word per
word not byte per byte, and even if the first bytes written could be at
the last address, I have never seen it done. However it is a good design
practice to explicitly separate the writing of the final CRC, to make it
obvious that you expect the previous write to be complete.
The design you described should work and inspires confidence.
Sebastien
Re: GSoC Final Report on mnemofs
Posted by Saurav Pal <re...@gmail.com>.
Hi all,
Thank you for the feedback and support.
Sebastien:
Thanks for the valuable points you have mentioned. Out of them, there are
some points that have been considered while designing this, and there are
some that can be used to improve it.
Theoretically, power loss should not be a problem. However, I agree that it
isn't a very concrete claim as it's yet to be tested out. This is mostly
due to a lack of actual drivers for the hardware at hand. The NAND sim I
had written helps simulate the structure of a NAND flash, and bad blocks,
but physical irregularities are a different matter. It does not yet
consider any form of bit flipping, nor does it have the capability to
simulate power losses yet.
Though, again theoretically, the worst that will happen in usual power loss
situations will be that all the changes stored on the LRU are gone.
The FS follows a copy on write system, and the old version of the file
system is always accessible till the new version is *completely written*.
However, there's one edge case that I'm worried about...where the pages in
the flash aren't enough in number to have both a new and old copy available
(which is what happens at one point during the update procedure). As for
the journal, it does ensure that the checksum is written *after* the
journal log. Assuming the bytes are written to the flash from the start to
end (which I have not seen anything to suggest otherwise), the checksum
will get written at the very end. However, thanks for the suggestion, I
will look more into the best way to apply checksums in this situation. For
now, the checksum is written along with the data, to lower wasted space (as
a separate write would need to be written onto a new page), but I will look
into this in more detail.
For application of these logs, the LRU and the journal are used together to
update the FS from the leaves (files) and go upwards. However, the old FS
still remains, and until the root of the FS is updated, the old one is the
only valid FS. When the new root is written, it will essentially be the
latest revision of the root, and the latest revision (along with a
checksum) is used as the root of the FS on mount in case of a power loss in
between the update. The partial completion of the update process can be
resumed again later on (of course, this is theoretical, but it should be
testable once the driver gets done, or the power loss feature gets added to
the NAND flash sim).
As for bit flipping, which is a more common occurrence in NAND flashes
because of the compact nature, the NAND driver takes care of bit flips.
There is a dedicated spare area that contains ECC bits for this purpose,
and the driver utilizes them before providing it to the FS. However, it
still has a limit for the amount of bits that can be rectified, and testing
in these situations is yet to be done.
I'll be a bit too busy till the end of October due to personal reasons, but
following that, I'll be spending time improving the FS. For now, it works
in theory, but the implementation of mnemofs still needs quite a bit of
work, as it was developed under a fairly short deadline required by GSoC,
which I plan to do.
Thanks a lot to all of you for this feedback, it highlighted some of the
various focal points a bit more, and all that I need to focus on to improve
this.
Best regards,
Saurav.
On Fri, Sep 13, 2024 at 6:57 PM Sebastien Lorquet <se...@lorquet.fr>
wrote:
> Hi,
>
> Yes before we claim having a power cut resilient FS , it has to be
> tested on real hardware.
>
> It will also be the occasion to actually fire-proof littlefs, because I
> have no idea if that was ever done before.
>
> We suggested the idea in a littlefs github issue but it got nowhere IIRC.
>
> Sebastien
>
> On 13/09/2024 15:20, Alan C. Assis wrote:
> > Hi Tomek,
> >
> > I think it is possible to modify the NAND Simulator to help testing
> > partial writes and induced errors.
> >
> > But I agree with Sebastien that it is better to test in real hardware.
> >
> > Since the MnemoFS already works in the simulator we can get the
> > initial version working in any flash, even SPI NOR Flash.
> >
> > BR,
> >
> > Alan
> >
> > On Fri, Sep 13, 2024 at 10:05 AM Tomek CEDRO <to...@cedro.info> wrote:
> >
> > very nice discussion!
> >
> > can problems with NAND create a semi-dereministic group of well
> > defined
> > issues with random characteristics?
> >
> > if so then we could create a model nand for sim with controllable
> > errors in
> > order to verify various nand drivers, filesystems, etc?
> >
> > :-)
> >
> > --
> > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >
> > On Fri, Sep 13, 2024, 14:52 Alan C. Assis <ac...@gmail.com> wrote:
> >
> > > Hi Sebastien,
> > >
> > > Thank you for your helpful considerations.
> > >
> > > As I explained before he used the SIM NAND Simulator that he
> > created and
> > > integrated on NuttX.
> > >
> > > Also as I explained in my previous email, we need help to test
> > in real
> > > hardware.
> > >
> > > Since you have previous experience with NAND Flash, maybe you
> > could help
> > > here (of course, if you are interested to help)!
> > >
> > > First we need to create a driver for a SPI NAND Flash (I bought
> > this model:
> > > https://aliexpress.com/item/1005005307786079.html) and use it with
> > > MnemoFS.
> > >
> > > This model that I selected has internal error detection, etc, it
> > means we
> > > don't need to worry about taking care of bad blocks ourselves.
> > >
> > > If you look inside nuttx/drivers/mtd/ many of the pieces we need
> are
> > > already there, we just need to understand how to use SPI NAND,
> > FTL, MTD,
> > > etc.
> > >
> > > Xiang, since you and your team ported YAFFS to NuttX, maybe you
> > guys could
> > > help us to get MnemoFS working on real flash on NuttX.
> > >
> > > BR,
> > >
> > > Alan
> > >
> > >
> > >
> > > On Fri, Sep 13, 2024 at 6:17 AM Sebastien Lorquet
> > <se...@lorquet.fr>
> > > wrote:
> > >
> > > > Hello
> > > >
> > > >
> > > > This is quite a complete report with a lot of details, this
> > shows that
> > > > you have put some large amount of mental energy in this
> > project, so
> > > > congratulations and thank you.
> > > >
> > > > What I'm about to write is not a critic but a complement that may
> > > > interest you.
> > > >
> > > >
> > > > Since I've worked with critical flash systems for more than 10
> > years
> > > > now, I have read the part of your document that deals with
> > power loss
> > > > with great interest.
> > > >
> > > > Resilience to power loss is *absolutely critical* to any embedded
> > > > filesystem.
> > > >
> > > >
> > > > Did you do power interruption tests on your code? Can you
> > guarantee that
> > > > the device format stays consistent/recoverable when the power
> > is cut at
> > > > any code location? Did you identify power critical code
> > sections (with
> > > > relation to power cut, not cpu access) ?
> > > >
> > > > Remember, if it's not tested, it doesnt work...
> > > >
> > > >
> > > > The most critical part of your work is the journal. Do you
> > make sure
> > > > that the checksum is written 1-last, and 2-completely? How do
> > you make
> > > > sure that the journal entries are correctly applied to their
> final
> > > > storage locations?
> > > >
> > > > The largest problem in that area is flash metastability. The
> > checksum
> > > > MIGHT appear correct on one read, but not correct at the next
> > access.
> > > > The reason for this is the analog nature of flash writes (and
> > erases),
> > > > which injects a number of electrons in a floating gate. 0 and
> > 1 bits are
> > > > separated by thresholds, but these thresholds vary with
> > temperature and
> > > > time (wear), so it might appear that a bit is correct by being
> > just at
> > > > the threshold, but the next access will result in a flipped bit.
> > > >
> > > > These issues are NOT theoretical, they happen all the time in
> > all flash
> > > > devices, you just have to tickle the devices often enough at
> > the right
> > > > moment so you begin to see these.
> > > >
> > > > These tests require the ability to fully cut the power to a
> > test board
> > > > with microsecond precision. No need for pulses, just an
> adjustable
> > > > delay. Test is triggered by a command that also start a
> > countdown, and
> > > > timeout is increased microsecond by microsecond until you
> > reach the
> > > > point that the flash is actually written. Usually, there is a
> > point
> > > > where timeouts result in partial writes. Then the board will
> start
> > > > acting funny and will start entering the error branches that
> > are usually
> > > > never taken. Board capacitors are not a problem, they just
> > increase the
> > > > delays. They always discharge the same way during all repeated
> > tests, so
> > > > they have no influence on the process.
> > > >
> > > > It is quite hard to make sure that everything is correct, but a
> > > > sufficient amount of dedication is required to be aware of the
> > potential
> > > > problems.
> > > >
> > > > How do you know in your filesystem that the checksum has been
> > written
> > > > only after all the previous data are written? How do you know the
> > > > checksum write is complete. There are software techniques for
> > this. This
> > > > also requires the flash to support overwrites, so making this
> > work with
> > > > ECC is harder (but possible).
> > > >
> > > > Fine details absolutely matters here.
> > > >
> > > > Thanks,
> > > >
> > > > Sebastien
> > > >
> > > >
> > > > On 12/09/2024 17:48, Saurav Pal wrote:
> > > > > Hi all,
> > > > >
> > > > > Here's my final report <
> > > https://resyfer.github.io/blogs/mnemofs/endeval/
> > > > >
> > > > > on mnemofs, a NAND flash file system for NuttX, on which I
> > worked
> > > during
> > > > my
> > > > > tenure as a GSoC 2024 Contributor for ASF. I would be
> > grateful for any
> > > > > suggestions and criticism.
> > > > >
> > > > > Best regards,
> > > > > Saurav Pal.
> > > > >
> > > >
> > >
> >
Re: GSoC Final Report on mnemofs
Posted by Sebastien Lorquet <se...@lorquet.fr>.
Hi,
Yes before we claim having a power cut resilient FS , it has to be
tested on real hardware.
It will also be the occasion to actually fire-proof littlefs, because I
have no idea if that was ever done before.
We suggested the idea in a littlefs github issue but it got nowhere IIRC.
Sebastien
On 13/09/2024 15:20, Alan C. Assis wrote:
> Hi Tomek,
>
> I think it is possible to modify the NAND Simulator to help testing
> partial writes and induced errors.
>
> But I agree with Sebastien that it is better to test in real hardware.
>
> Since the MnemoFS already works in the simulator we can get the
> initial version working in any flash, even SPI NOR Flash.
>
> BR,
>
> Alan
>
> On Fri, Sep 13, 2024 at 10:05 AM Tomek CEDRO <to...@cedro.info> wrote:
>
> very nice discussion!
>
> can problems with NAND create a semi-dereministic group of well
> defined
> issues with random characteristics?
>
> if so then we could create a model nand for sim with controllable
> errors in
> order to verify various nand drivers, filesystems, etc?
>
> :-)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>
> On Fri, Sep 13, 2024, 14:52 Alan C. Assis <ac...@gmail.com> wrote:
>
> > Hi Sebastien,
> >
> > Thank you for your helpful considerations.
> >
> > As I explained before he used the SIM NAND Simulator that he
> created and
> > integrated on NuttX.
> >
> > Also as I explained in my previous email, we need help to test
> in real
> > hardware.
> >
> > Since you have previous experience with NAND Flash, maybe you
> could help
> > here (of course, if you are interested to help)!
> >
> > First we need to create a driver for a SPI NAND Flash (I bought
> this model:
> > https://aliexpress.com/item/1005005307786079.html) and use it with
> > MnemoFS.
> >
> > This model that I selected has internal error detection, etc, it
> means we
> > don't need to worry about taking care of bad blocks ourselves.
> >
> > If you look inside nuttx/drivers/mtd/ many of the pieces we need are
> > already there, we just need to understand how to use SPI NAND,
> FTL, MTD,
> > etc.
> >
> > Xiang, since you and your team ported YAFFS to NuttX, maybe you
> guys could
> > help us to get MnemoFS working on real flash on NuttX.
> >
> > BR,
> >
> > Alan
> >
> >
> >
> > On Fri, Sep 13, 2024 at 6:17 AM Sebastien Lorquet
> <se...@lorquet.fr>
> > wrote:
> >
> > > Hello
> > >
> > >
> > > This is quite a complete report with a lot of details, this
> shows that
> > > you have put some large amount of mental energy in this
> project, so
> > > congratulations and thank you.
> > >
> > > What I'm about to write is not a critic but a complement that may
> > > interest you.
> > >
> > >
> > > Since I've worked with critical flash systems for more than 10
> years
> > > now, I have read the part of your document that deals with
> power loss
> > > with great interest.
> > >
> > > Resilience to power loss is *absolutely critical* to any embedded
> > > filesystem.
> > >
> > >
> > > Did you do power interruption tests on your code? Can you
> guarantee that
> > > the device format stays consistent/recoverable when the power
> is cut at
> > > any code location? Did you identify power critical code
> sections (with
> > > relation to power cut, not cpu access) ?
> > >
> > > Remember, if it's not tested, it doesnt work...
> > >
> > >
> > > The most critical part of your work is the journal. Do you
> make sure
> > > that the checksum is written 1-last, and 2-completely? How do
> you make
> > > sure that the journal entries are correctly applied to their final
> > > storage locations?
> > >
> > > The largest problem in that area is flash metastability. The
> checksum
> > > MIGHT appear correct on one read, but not correct at the next
> access.
> > > The reason for this is the analog nature of flash writes (and
> erases),
> > > which injects a number of electrons in a floating gate. 0 and
> 1 bits are
> > > separated by thresholds, but these thresholds vary with
> temperature and
> > > time (wear), so it might appear that a bit is correct by being
> just at
> > > the threshold, but the next access will result in a flipped bit.
> > >
> > > These issues are NOT theoretical, they happen all the time in
> all flash
> > > devices, you just have to tickle the devices often enough at
> the right
> > > moment so you begin to see these.
> > >
> > > These tests require the ability to fully cut the power to a
> test board
> > > with microsecond precision. No need for pulses, just an adjustable
> > > delay. Test is triggered by a command that also start a
> countdown, and
> > > timeout is increased microsecond by microsecond until you
> reach the
> > > point that the flash is actually written. Usually, there is a
> point
> > > where timeouts result in partial writes. Then the board will start
> > > acting funny and will start entering the error branches that
> are usually
> > > never taken. Board capacitors are not a problem, they just
> increase the
> > > delays. They always discharge the same way during all repeated
> tests, so
> > > they have no influence on the process.
> > >
> > > It is quite hard to make sure that everything is correct, but a
> > > sufficient amount of dedication is required to be aware of the
> potential
> > > problems.
> > >
> > > How do you know in your filesystem that the checksum has been
> written
> > > only after all the previous data are written? How do you know the
> > > checksum write is complete. There are software techniques for
> this. This
> > > also requires the flash to support overwrites, so making this
> work with
> > > ECC is harder (but possible).
> > >
> > > Fine details absolutely matters here.
> > >
> > > Thanks,
> > >
> > > Sebastien
> > >
> > >
> > > On 12/09/2024 17:48, Saurav Pal wrote:
> > > > Hi all,
> > > >
> > > > Here's my final report <
> > https://resyfer.github.io/blogs/mnemofs/endeval/
> > > >
> > > > on mnemofs, a NAND flash file system for NuttX, on which I
> worked
> > during
> > > my
> > > > tenure as a GSoC 2024 Contributor for ASF. I would be
> grateful for any
> > > > suggestions and criticism.
> > > >
> > > > Best regards,
> > > > Saurav Pal.
> > > >
> > >
> >
>
Re: GSoC Final Report on mnemofs
Posted by "Alan C. Assis" <ac...@gmail.com>.
Hi Tomek,
I think it is possible to modify the NAND Simulator to help testing partial
writes and induced errors.
But I agree with Sebastien that it is better to test in real hardware.
Since the MnemoFS already works in the simulator we can get the initial
version working in any flash, even SPI NOR Flash.
BR,
Alan
On Fri, Sep 13, 2024 at 10:05 AM Tomek CEDRO <to...@cedro.info> wrote:
> very nice discussion!
>
> can problems with NAND create a semi-dereministic group of well defined
> issues with random characteristics?
>
> if so then we could create a model nand for sim with controllable errors in
> order to verify various nand drivers, filesystems, etc?
>
> :-)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>
> On Fri, Sep 13, 2024, 14:52 Alan C. Assis <ac...@gmail.com> wrote:
>
> > Hi Sebastien,
> >
> > Thank you for your helpful considerations.
> >
> > As I explained before he used the SIM NAND Simulator that he created and
> > integrated on NuttX.
> >
> > Also as I explained in my previous email, we need help to test in real
> > hardware.
> >
> > Since you have previous experience with NAND Flash, maybe you could help
> > here (of course, if you are interested to help)!
> >
> > First we need to create a driver for a SPI NAND Flash (I bought this
> model:
> > https://aliexpress.com/item/1005005307786079.html) and use it with
> > MnemoFS.
> >
> > This model that I selected has internal error detection, etc, it means we
> > don't need to worry about taking care of bad blocks ourselves.
> >
> > If you look inside nuttx/drivers/mtd/ many of the pieces we need are
> > already there, we just need to understand how to use SPI NAND, FTL, MTD,
> > etc.
> >
> > Xiang, since you and your team ported YAFFS to NuttX, maybe you guys
> could
> > help us to get MnemoFS working on real flash on NuttX.
> >
> > BR,
> >
> > Alan
> >
> >
> >
> > On Fri, Sep 13, 2024 at 6:17 AM Sebastien Lorquet <se...@lorquet.fr>
> > wrote:
> >
> > > Hello
> > >
> > >
> > > This is quite a complete report with a lot of details, this shows that
> > > you have put some large amount of mental energy in this project, so
> > > congratulations and thank you.
> > >
> > > What I'm about to write is not a critic but a complement that may
> > > interest you.
> > >
> > >
> > > Since I've worked with critical flash systems for more than 10 years
> > > now, I have read the part of your document that deals with power loss
> > > with great interest.
> > >
> > > Resilience to power loss is *absolutely critical* to any embedded
> > > filesystem.
> > >
> > >
> > > Did you do power interruption tests on your code? Can you guarantee
> that
> > > the device format stays consistent/recoverable when the power is cut at
> > > any code location? Did you identify power critical code sections (with
> > > relation to power cut, not cpu access) ?
> > >
> > > Remember, if it's not tested, it doesnt work...
> > >
> > >
> > > The most critical part of your work is the journal. Do you make sure
> > > that the checksum is written 1-last, and 2-completely? How do you make
> > > sure that the journal entries are correctly applied to their final
> > > storage locations?
> > >
> > > The largest problem in that area is flash metastability. The checksum
> > > MIGHT appear correct on one read, but not correct at the next access.
> > > The reason for this is the analog nature of flash writes (and erases),
> > > which injects a number of electrons in a floating gate. 0 and 1 bits
> are
> > > separated by thresholds, but these thresholds vary with temperature and
> > > time (wear), so it might appear that a bit is correct by being just at
> > > the threshold, but the next access will result in a flipped bit.
> > >
> > > These issues are NOT theoretical, they happen all the time in all flash
> > > devices, you just have to tickle the devices often enough at the right
> > > moment so you begin to see these.
> > >
> > > These tests require the ability to fully cut the power to a test board
> > > with microsecond precision. No need for pulses, just an adjustable
> > > delay. Test is triggered by a command that also start a countdown, and
> > > timeout is increased microsecond by microsecond until you reach the
> > > point that the flash is actually written. Usually, there is a point
> > > where timeouts result in partial writes. Then the board will start
> > > acting funny and will start entering the error branches that are
> usually
> > > never taken. Board capacitors are not a problem, they just increase the
> > > delays. They always discharge the same way during all repeated tests,
> so
> > > they have no influence on the process.
> > >
> > > It is quite hard to make sure that everything is correct, but a
> > > sufficient amount of dedication is required to be aware of the
> potential
> > > problems.
> > >
> > > How do you know in your filesystem that the checksum has been written
> > > only after all the previous data are written? How do you know the
> > > checksum write is complete. There are software techniques for this.
> This
> > > also requires the flash to support overwrites, so making this work with
> > > ECC is harder (but possible).
> > >
> > > Fine details absolutely matters here.
> > >
> > > Thanks,
> > >
> > > Sebastien
> > >
> > >
> > > On 12/09/2024 17:48, Saurav Pal wrote:
> > > > Hi all,
> > > >
> > > > Here's my final report <
> > https://resyfer.github.io/blogs/mnemofs/endeval/
> > > >
> > > > on mnemofs, a NAND flash file system for NuttX, on which I worked
> > during
> > > my
> > > > tenure as a GSoC 2024 Contributor for ASF. I would be grateful for
> any
> > > > suggestions and criticism.
> > > >
> > > > Best regards,
> > > > Saurav Pal.
> > > >
> > >
> >
>
Re: GSoC Final Report on mnemofs
Posted by Sebastien Lorquet <se...@lorquet.fr>.
Hi,
No idea, Tomek! flash works by storing charge (electrons) in a floating
gate, like you fill a bottle with water. Empty (erased) bottle means one
and above a certain level it becomes zero.
Problem is, the bottle material has a temperature coefficient so it
becomes larger when hot.
My feeling is that thresholds effect WILL appear given the proper delay
for the power cut, you cant avoid them. We encounter these regularly
during development of our products and we have explicit tests for them.
Alan: I dont know if my help will be efficient since I have so many
things pending.
For NAND you will probably map all functions to MTD and error correction
features to ioctls. Many of these will vary according to the model.
For variation, I have added these Micron SLC NAND devices to my next
farnell order:
https://fr.farnell.com/micron/mt29f1g01abafdwb-it-f/flash-memory-1gbit-40-to-85deg/dp/3954455
https://fr.farnell.com/micron/mt29f4g01abafdwb-it-f/flash-memory-4gbit-40-to-85deg/dp/3954460
These are UPDFN packages, which are the same size than a SOIC 8.
And wow, these devices have much larger capacity than NOR flash!
I have zero experience with NAND, but I can run code on some boards. For
example I can try that on a nucleo board, or on a pi pico.
Next we need some kind of system to control a delay with microsecond
precision, in order to cut power of the board (or do a RESET).
That kind of test will also help test other file systems like littlefs
on NOR flash, which has the same issues.
I will not be shy that help with software would be welcome. I can
execute things on hardware but I wont lead the project because I just
cant. And I mean my brain cant.
Sebastien
On 13/09/2024 15:02, Tomek CEDRO wrote:
> very nice discussion!
>
> can problems with NAND create a semi-dereministic group of well
> defined issues with random characteristics?
>
> if so then we could create a model nand for sim with controllable
> errors in order to verify various nand drivers, filesystems, etc?
>
> :-)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>
> On Fri, Sep 13, 2024, 14:52 Alan C. Assis <ac...@gmail.com> wrote:
>
> Hi Sebastien,
>
> Thank you for your helpful considerations.
>
> As I explained before he used the SIM NAND Simulator that he
> created and
> integrated on NuttX.
>
> Also as I explained in my previous email, we need help to test in real
> hardware.
>
> Since you have previous experience with NAND Flash, maybe you
> could help
> here (of course, if you are interested to help)!
>
> First we need to create a driver for a SPI NAND Flash (I bought
> this model:
> https://aliexpress.com/item/1005005307786079.html) and use it with
> MnemoFS.
>
> This model that I selected has internal error detection, etc, it
> means we
> don't need to worry about taking care of bad blocks ourselves.
>
> If you look inside nuttx/drivers/mtd/ many of the pieces we need are
> already there, we just need to understand how to use SPI NAND,
> FTL, MTD,
> etc.
>
> Xiang, since you and your team ported YAFFS to NuttX, maybe you
> guys could
> help us to get MnemoFS working on real flash on NuttX.
>
> BR,
>
> Alan
>
>
>
> On Fri, Sep 13, 2024 at 6:17 AM Sebastien Lorquet
> <se...@lorquet.fr>
> wrote:
>
> > Hello
> >
> >
> > This is quite a complete report with a lot of details, this
> shows that
> > you have put some large amount of mental energy in this project, so
> > congratulations and thank you.
> >
> > What I'm about to write is not a critic but a complement that may
> > interest you.
> >
> >
> > Since I've worked with critical flash systems for more than 10 years
> > now, I have read the part of your document that deals with power
> loss
> > with great interest.
> >
> > Resilience to power loss is *absolutely critical* to any embedded
> > filesystem.
> >
> >
> > Did you do power interruption tests on your code? Can you
> guarantee that
> > the device format stays consistent/recoverable when the power is
> cut at
> > any code location? Did you identify power critical code sections
> (with
> > relation to power cut, not cpu access) ?
> >
> > Remember, if it's not tested, it doesnt work...
> >
> >
> > The most critical part of your work is the journal. Do you make sure
> > that the checksum is written 1-last, and 2-completely? How do
> you make
> > sure that the journal entries are correctly applied to their final
> > storage locations?
> >
> > The largest problem in that area is flash metastability. The
> checksum
> > MIGHT appear correct on one read, but not correct at the next
> access.
> > The reason for this is the analog nature of flash writes (and
> erases),
> > which injects a number of electrons in a floating gate. 0 and 1
> bits are
> > separated by thresholds, but these thresholds vary with
> temperature and
> > time (wear), so it might appear that a bit is correct by being
> just at
> > the threshold, but the next access will result in a flipped bit.
> >
> > These issues are NOT theoretical, they happen all the time in
> all flash
> > devices, you just have to tickle the devices often enough at the
> right
> > moment so you begin to see these.
> >
> > These tests require the ability to fully cut the power to a test
> board
> > with microsecond precision. No need for pulses, just an adjustable
> > delay. Test is triggered by a command that also start a
> countdown, and
> > timeout is increased microsecond by microsecond until you reach the
> > point that the flash is actually written. Usually, there is a point
> > where timeouts result in partial writes. Then the board will start
> > acting funny and will start entering the error branches that are
> usually
> > never taken. Board capacitors are not a problem, they just
> increase the
> > delays. They always discharge the same way during all repeated
> tests, so
> > they have no influence on the process.
> >
> > It is quite hard to make sure that everything is correct, but a
> > sufficient amount of dedication is required to be aware of the
> potential
> > problems.
> >
> > How do you know in your filesystem that the checksum has been
> written
> > only after all the previous data are written? How do you know the
> > checksum write is complete. There are software techniques for
> this. This
> > also requires the flash to support overwrites, so making this
> work with
> > ECC is harder (but possible).
> >
> > Fine details absolutely matters here.
> >
> > Thanks,
> >
> > Sebastien
> >
> >
> > On 12/09/2024 17:48, Saurav Pal wrote:
> > > Hi all,
> > >
> > > Here's my final report
> <https://resyfer.github.io/blogs/mnemofs/endeval/
> > >
> > > on mnemofs, a NAND flash file system for NuttX, on which I
> worked during
> > my
> > > tenure as a GSoC 2024 Contributor for ASF. I would be grateful
> for any
> > > suggestions and criticism.
> > >
> > > Best regards,
> > > Saurav Pal.
> > >
> >
>
Re: GSoC Final Report on mnemofs
Posted by Tomek CEDRO <to...@cedro.info>.
very nice discussion!
can problems with NAND create a semi-dereministic group of well defined
issues with random characteristics?
if so then we could create a model nand for sim with controllable errors in
order to verify various nand drivers, filesystems, etc?
:-)
--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
On Fri, Sep 13, 2024, 14:52 Alan C. Assis <ac...@gmail.com> wrote:
> Hi Sebastien,
>
> Thank you for your helpful considerations.
>
> As I explained before he used the SIM NAND Simulator that he created and
> integrated on NuttX.
>
> Also as I explained in my previous email, we need help to test in real
> hardware.
>
> Since you have previous experience with NAND Flash, maybe you could help
> here (of course, if you are interested to help)!
>
> First we need to create a driver for a SPI NAND Flash (I bought this model:
> https://aliexpress.com/item/1005005307786079.html) and use it with
> MnemoFS.
>
> This model that I selected has internal error detection, etc, it means we
> don't need to worry about taking care of bad blocks ourselves.
>
> If you look inside nuttx/drivers/mtd/ many of the pieces we need are
> already there, we just need to understand how to use SPI NAND, FTL, MTD,
> etc.
>
> Xiang, since you and your team ported YAFFS to NuttX, maybe you guys could
> help us to get MnemoFS working on real flash on NuttX.
>
> BR,
>
> Alan
>
>
>
> On Fri, Sep 13, 2024 at 6:17 AM Sebastien Lorquet <se...@lorquet.fr>
> wrote:
>
> > Hello
> >
> >
> > This is quite a complete report with a lot of details, this shows that
> > you have put some large amount of mental energy in this project, so
> > congratulations and thank you.
> >
> > What I'm about to write is not a critic but a complement that may
> > interest you.
> >
> >
> > Since I've worked with critical flash systems for more than 10 years
> > now, I have read the part of your document that deals with power loss
> > with great interest.
> >
> > Resilience to power loss is *absolutely critical* to any embedded
> > filesystem.
> >
> >
> > Did you do power interruption tests on your code? Can you guarantee that
> > the device format stays consistent/recoverable when the power is cut at
> > any code location? Did you identify power critical code sections (with
> > relation to power cut, not cpu access) ?
> >
> > Remember, if it's not tested, it doesnt work...
> >
> >
> > The most critical part of your work is the journal. Do you make sure
> > that the checksum is written 1-last, and 2-completely? How do you make
> > sure that the journal entries are correctly applied to their final
> > storage locations?
> >
> > The largest problem in that area is flash metastability. The checksum
> > MIGHT appear correct on one read, but not correct at the next access.
> > The reason for this is the analog nature of flash writes (and erases),
> > which injects a number of electrons in a floating gate. 0 and 1 bits are
> > separated by thresholds, but these thresholds vary with temperature and
> > time (wear), so it might appear that a bit is correct by being just at
> > the threshold, but the next access will result in a flipped bit.
> >
> > These issues are NOT theoretical, they happen all the time in all flash
> > devices, you just have to tickle the devices often enough at the right
> > moment so you begin to see these.
> >
> > These tests require the ability to fully cut the power to a test board
> > with microsecond precision. No need for pulses, just an adjustable
> > delay. Test is triggered by a command that also start a countdown, and
> > timeout is increased microsecond by microsecond until you reach the
> > point that the flash is actually written. Usually, there is a point
> > where timeouts result in partial writes. Then the board will start
> > acting funny and will start entering the error branches that are usually
> > never taken. Board capacitors are not a problem, they just increase the
> > delays. They always discharge the same way during all repeated tests, so
> > they have no influence on the process.
> >
> > It is quite hard to make sure that everything is correct, but a
> > sufficient amount of dedication is required to be aware of the potential
> > problems.
> >
> > How do you know in your filesystem that the checksum has been written
> > only after all the previous data are written? How do you know the
> > checksum write is complete. There are software techniques for this. This
> > also requires the flash to support overwrites, so making this work with
> > ECC is harder (but possible).
> >
> > Fine details absolutely matters here.
> >
> > Thanks,
> >
> > Sebastien
> >
> >
> > On 12/09/2024 17:48, Saurav Pal wrote:
> > > Hi all,
> > >
> > > Here's my final report <
> https://resyfer.github.io/blogs/mnemofs/endeval/
> > >
> > > on mnemofs, a NAND flash file system for NuttX, on which I worked
> during
> > my
> > > tenure as a GSoC 2024 Contributor for ASF. I would be grateful for any
> > > suggestions and criticism.
> > >
> > > Best regards,
> > > Saurav Pal.
> > >
> >
>
Re: GSoC Final Report on mnemofs
Posted by "Alan C. Assis" <ac...@gmail.com>.
Hi Sebastien,
Thank you for your helpful considerations.
As I explained before he used the SIM NAND Simulator that he created and
integrated on NuttX.
Also as I explained in my previous email, we need help to test in real
hardware.
Since you have previous experience with NAND Flash, maybe you could help
here (of course, if you are interested to help)!
First we need to create a driver for a SPI NAND Flash (I bought this model:
https://aliexpress.com/item/1005005307786079.html) and use it with MnemoFS.
This model that I selected has internal error detection, etc, it means we
don't need to worry about taking care of bad blocks ourselves.
If you look inside nuttx/drivers/mtd/ many of the pieces we need are
already there, we just need to understand how to use SPI NAND, FTL, MTD,
etc.
Xiang, since you and your team ported YAFFS to NuttX, maybe you guys could
help us to get MnemoFS working on real flash on NuttX.
BR,
Alan
On Fri, Sep 13, 2024 at 6:17 AM Sebastien Lorquet <se...@lorquet.fr>
wrote:
> Hello
>
>
> This is quite a complete report with a lot of details, this shows that
> you have put some large amount of mental energy in this project, so
> congratulations and thank you.
>
> What I'm about to write is not a critic but a complement that may
> interest you.
>
>
> Since I've worked with critical flash systems for more than 10 years
> now, I have read the part of your document that deals with power loss
> with great interest.
>
> Resilience to power loss is *absolutely critical* to any embedded
> filesystem.
>
>
> Did you do power interruption tests on your code? Can you guarantee that
> the device format stays consistent/recoverable when the power is cut at
> any code location? Did you identify power critical code sections (with
> relation to power cut, not cpu access) ?
>
> Remember, if it's not tested, it doesnt work...
>
>
> The most critical part of your work is the journal. Do you make sure
> that the checksum is written 1-last, and 2-completely? How do you make
> sure that the journal entries are correctly applied to their final
> storage locations?
>
> The largest problem in that area is flash metastability. The checksum
> MIGHT appear correct on one read, but not correct at the next access.
> The reason for this is the analog nature of flash writes (and erases),
> which injects a number of electrons in a floating gate. 0 and 1 bits are
> separated by thresholds, but these thresholds vary with temperature and
> time (wear), so it might appear that a bit is correct by being just at
> the threshold, but the next access will result in a flipped bit.
>
> These issues are NOT theoretical, they happen all the time in all flash
> devices, you just have to tickle the devices often enough at the right
> moment so you begin to see these.
>
> These tests require the ability to fully cut the power to a test board
> with microsecond precision. No need for pulses, just an adjustable
> delay. Test is triggered by a command that also start a countdown, and
> timeout is increased microsecond by microsecond until you reach the
> point that the flash is actually written. Usually, there is a point
> where timeouts result in partial writes. Then the board will start
> acting funny and will start entering the error branches that are usually
> never taken. Board capacitors are not a problem, they just increase the
> delays. They always discharge the same way during all repeated tests, so
> they have no influence on the process.
>
> It is quite hard to make sure that everything is correct, but a
> sufficient amount of dedication is required to be aware of the potential
> problems.
>
> How do you know in your filesystem that the checksum has been written
> only after all the previous data are written? How do you know the
> checksum write is complete. There are software techniques for this. This
> also requires the flash to support overwrites, so making this work with
> ECC is harder (but possible).
>
> Fine details absolutely matters here.
>
> Thanks,
>
> Sebastien
>
>
> On 12/09/2024 17:48, Saurav Pal wrote:
> > Hi all,
> >
> > Here's my final report <https://resyfer.github.io/blogs/mnemofs/endeval/
> >
> > on mnemofs, a NAND flash file system for NuttX, on which I worked during
> my
> > tenure as a GSoC 2024 Contributor for ASF. I would be grateful for any
> > suggestions and criticism.
> >
> > Best regards,
> > Saurav Pal.
> >
>
Re: GSoC Final Report on mnemofs
Posted by Sebastien Lorquet <se...@lorquet.fr>.
Hello
This is quite a complete report with a lot of details, this shows that
you have put some large amount of mental energy in this project, so
congratulations and thank you.
What I'm about to write is not a critic but a complement that may
interest you.
Since I've worked with critical flash systems for more than 10 years
now, I have read the part of your document that deals with power loss
with great interest.
Resilience to power loss is *absolutely critical* to any embedded
filesystem.
Did you do power interruption tests on your code? Can you guarantee that
the device format stays consistent/recoverable when the power is cut at
any code location? Did you identify power critical code sections (with
relation to power cut, not cpu access) ?
Remember, if it's not tested, it doesnt work...
The most critical part of your work is the journal. Do you make sure
that the checksum is written 1-last, and 2-completely? How do you make
sure that the journal entries are correctly applied to their final
storage locations?
The largest problem in that area is flash metastability. The checksum
MIGHT appear correct on one read, but not correct at the next access.
The reason for this is the analog nature of flash writes (and erases),
which injects a number of electrons in a floating gate. 0 and 1 bits are
separated by thresholds, but these thresholds vary with temperature and
time (wear), so it might appear that a bit is correct by being just at
the threshold, but the next access will result in a flipped bit.
These issues are NOT theoretical, they happen all the time in all flash
devices, you just have to tickle the devices often enough at the right
moment so you begin to see these.
These tests require the ability to fully cut the power to a test board
with microsecond precision. No need for pulses, just an adjustable
delay. Test is triggered by a command that also start a countdown, and
timeout is increased microsecond by microsecond until you reach the
point that the flash is actually written. Usually, there is a point
where timeouts result in partial writes. Then the board will start
acting funny and will start entering the error branches that are usually
never taken. Board capacitors are not a problem, they just increase the
delays. They always discharge the same way during all repeated tests, so
they have no influence on the process.
It is quite hard to make sure that everything is correct, but a
sufficient amount of dedication is required to be aware of the potential
problems.
How do you know in your filesystem that the checksum has been written
only after all the previous data are written? How do you know the
checksum write is complete. There are software techniques for this. This
also requires the flash to support overwrites, so making this work with
ECC is harder (but possible).
Fine details absolutely matters here.
Thanks,
Sebastien
On 12/09/2024 17:48, Saurav Pal wrote:
> Hi all,
>
> Here's my final report <https://resyfer.github.io/blogs/mnemofs/endeval/>
> on mnemofs, a NAND flash file system for NuttX, on which I worked during my
> tenure as a GSoC 2024 Contributor for ASF. I would be grateful for any
> suggestions and criticism.
>
> Best regards,
> Saurav Pal.
>