You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mynewt.apache.org by Fabio Utzig <ut...@utzig.org> on 2017/01/13 10:43:12 UTC
Supporting multiple filesystems and "block" devices
Hello,
I've been working on patch that adds support for accessing multiple
devices and filesystems on Mynewt (overcoming the current single FS and
"hardcoded" block device per driver). The PR is available here:
https://github.com/apache/incubator-mynewt-core/pull/158
This PR is completely functional with some limitations which I'll talk
about below and plan to add on a later PR (given the current one is
already a lot of changes).
The main visible change is that one new package is added which is
responsible to track what disks use what filesystems. I called it
"fs/disk". From a user perspective there is an extra step required which
is a call like this:
disk_register("mmc0", "fatfs", &mmc_ops);
disk_register("flash0", "nffs", &flash_ops);
This adds a name for the disk, what FS is used (FS's are registered when
loaded), and what "disk" ops should be used for that named device. The
disk_ops struct is basically an abstraction of possible operations a
disk can do like read/write/erase/etc.
All the fs_* functions where updated to support multiple disks. To
access a file/dir now will required adding the name of the disk as a
prefix like this:
rc = fs_opendir("mmc0:/", &dir);
rc = fs_open("flash0:/testfile.txt", FS_ACCESS_READ, &file);
That "mmc0:" and "flash0:" should be the same names that were used when
registering. Of course, one can always use fatfs_open, nffs_open
directly and not need to specify the disk. Also for compatibility
reasons, when only a single FS is used on the project (single FS dep)
the disk prefix is optional (to not break existing code, tests, etc).
But the disk_register must be called nonetheless.
What still needs to be done?
1) Currently FAT is updated to use disk_ops because it only uses
read/write but it should work on other block devices (would, of course,
fail on a real flash that requires erase!). NFFS is still calling the
hal_flash_ functions and should be updated to use the disk_ops interface
(as soon as it has extra calls for erase, etc).
2) Also previously discussed in pvt was the fact that we need to pass
some device context to low-level hal_flash (also mmc_) functions. I
recently wrote a SPI flash driver for AT45DB which will follow the
hal_flash_ interface but it needs to receive an extra context to know
which SPI to use, which configuration, etc. This will probably also add
another parameter to disk_register with the said context struct.
I'll be fixing the above issues on a new PR.
Please review the current PR, add suggestions, etc.
Cheers,
Fabio Utzig
Re: Supporting multiple filesystems and "block" devices
Posted by Fabio Utzig <ut...@utzig.org>.
On Fri, Jan 13, 2017, at 08:21 PM, David G. Simmons wrote:
> +1 on UNIX paths. I have PTSD from DOS paths. :-)
Can't really be called DOS path if it uses a string of any length (yup,
followed by ':') to name drives and '/' as directory separator! :-)
Re: Supporting multiple filesystems and "block" devices
Posted by "David G. Simmons" <sa...@mac.com>.
+1 on UNIX paths. I have PTSD from DOS paths. :-)
dg
> On Jan 13, 2017, at 5:07 PM, Christopher Collins <cc...@apache.org> wrote:
>
> Hi Fabio,
>
> Sounds great. I only have one comment:
>
> On Fri, Jan 13, 2017 at 03:09:48PM -0200, Fabio Utzig wrote:
>>> All the fs_* functions where updated to support multiple disks. To
>>> access a file/dir now will required adding the name of the disk as a
>>> prefix like this:
>>>
>>> rc = fs_opendir("mmc0:/", &dir);
>>> rc = fs_open("flash0:/testfile.txt", FS_ACCESS_READ, &file);
>
> I'm sure you anticipated this comment :). Seeing a ':' in a path brings
> back horrible memories for me. I am not sure if this is just an
> aesthetic judgment, or if the "UNIX way" would actually be better. If
> it's just an asthetic thing, then I'll get over it, but I can't help but
> think there is code out there that would manage better with UNIX paths
> rather than DOS paths.
>
> I haven't looked at the code yet, so I don't know how much easier the
> use of the colon character makes things. What do you think about using
> "/flash0/testfile.txt" rather than "flash0/textfile.txt"?
>
> Chris
--
David G. Simmons
(919) 534-5099
Web <https://davidgs.com/> • Blog <https://davidgs.com/davidgs_blog> • Linkedin <http://linkedin.com/in/davidgsimmons> • Twitter <http://twitter.com/TechEvangelist1> • GitHub <http://github.com/davidgs>
/** Message digitally signed for security and authenticity.
* If you cannot read the PGP.sig attachment, please go to
* http://www.gnupg.com/ <http://www.gnupg.com/> Secure your email!!!
* Public key available at keyserver.pgp.com <http://keyserver.pgp.com/>
**/
♺ This email uses 100% recycled electrons. Don't blow it by printing!
There are only 2 hard things in computer science: Cache invalidation, naming things, and off-by-one errors.
Re: Supporting multiple filesystems and "block" devices
Posted by Christopher Collins <cc...@apache.org>.
On Fri, Jan 13, 2017 at 08:29:33PM -0200, Fabio Utzig wrote:
> Yes, I was expecting it! So I had 3 options in mind:
>
> Option 1 - Use a "drive" name as was implemented. Btw, it's not limited
> to single letter drive names! Sure if someone really likes DOS, they
> could use A:, B:, etc. This choice is extremely easy to parse. And it
> also brings me fond memories of AmigaDOS!
>
> Option 2 - Use a directory as a mount point like Unix but without
> implementing a full tree struct. Also very easy to parse, but somehow I
> find it very non-elegant because there would be no root path and such. I
> just personally dislike it because it looks incomplete, which brings me
> to:
>
> Option 3 - Use a proper tree structure with mount points, mounts inside
> other mount's directories, etc. This is quite harder to do properly but
> would be my preferred aesthetic choice. But this needs extra things like
> ensuring the ordering of mounts, more error handling (if a root mount
> fails for any reason what to do with the remaining drives, etc). I would
> like to move to this but it could be added later because it's basically
> adding code and what is in place stays there, I think.
>
> If you think a tree structure is a must and "drive" names are an
> aberration, I would be OK at giving a try on implementing a proper VFS!
Re: option 2- I understand what you mean about the missing root path,
and I agree that it would be odd. Personally, I don't think option 3 is
worth the effort, at least not now. Just speaking for myself, I think
option 1 is not a real problem. Like I said, it was just a knee-jerk
reaction when I saw the colon!
Thanks,
Chris
Re: Supporting multiple filesystems and "block" devices
Posted by Fabio Utzig <ut...@utzig.org>.
> I'm sure you anticipated this comment :). Seeing a ':' in a path brings
> back horrible memories for me. I am not sure if this is just an
> aesthetic judgment, or if the "UNIX way" would actually be better. If
> it's just an asthetic thing, then I'll get over it, but I can't help but
> think there is code out there that would manage better with UNIX paths
> rather than DOS paths.
>
> I haven't looked at the code yet, so I don't know how much easier the
> use of the colon character makes things. What do you think about using
> "/flash0/testfile.txt" rather than "flash0/textfile.txt"?
Yes, I was expecting it! So I had 3 options in mind:
Option 1 - Use a "drive" name as was implemented. Btw, it's not limited
to single letter drive names! Sure if someone really likes DOS, they
could use A:, B:, etc. This choice is extremely easy to parse. And it
also brings me fond memories of AmigaDOS!
Option 2 - Use a directory as a mount point like Unix but without
implementing a full tree struct. Also very easy to parse, but somehow I
find it very non-elegant because there would be no root path and such. I
just personally dislike it because it looks incomplete, which brings me
to:
Option 3 - Use a proper tree structure with mount points, mounts inside
other mount's directories, etc. This is quite harder to do properly but
would be my preferred aesthetic choice. But this needs extra things like
ensuring the ordering of mounts, more error handling (if a root mount
fails for any reason what to do with the remaining drives, etc). I would
like to move to this but it could be added later because it's basically
adding code and what is in place stays there, I think.
If you think a tree structure is a must and "drive" names are an
aberration, I would be OK at giving a try on implementing a proper VFS!
Cheers,
Fabio Utzig
Re: Supporting multiple filesystems and "block" devices
Posted by Christopher Collins <cc...@apache.org>.
Hi Fabio,
Sounds great. I only have one comment:
On Fri, Jan 13, 2017 at 03:09:48PM -0200, Fabio Utzig wrote:
> > All the fs_* functions where updated to support multiple disks. To
> > access a file/dir now will required adding the name of the disk as a
> > prefix like this:
> >
> > rc = fs_opendir("mmc0:/", &dir);
> > rc = fs_open("flash0:/testfile.txt", FS_ACCESS_READ, &file);
I'm sure you anticipated this comment :). Seeing a ':' in a path brings
back horrible memories for me. I am not sure if this is just an
aesthetic judgment, or if the "UNIX way" would actually be better. If
it's just an asthetic thing, then I'll get over it, but I can't help but
think there is code out there that would manage better with UNIX paths
rather than DOS paths.
I haven't looked at the code yet, so I don't know how much easier the
use of the colon character makes things. What do you think about using
"/flash0/testfile.txt" rather than "flash0/textfile.txt"?
Chris
Re: Supporting multiple filesystems and "block" devices
Posted by Kevin Townsend <ke...@adafruit.com>.
I think this makes a lot of sense. I don't have any strong opinions
about the device naming ... though I'm also coming from a fond Amiga BG
as well :) ... but having a generic abstraction between NFFS in system
flash, FatFS on SD cards over SPI or SDIO, who knows what on SPI flash,
etc., makes a lot of sense.
Pre 1.0 is a good time to get it in there, even if this is really a late
last minute addition.
I really like NFFS but FatFS is still the most useful way to get things
off a device to the outside world with a cheap uSD card, so it would be
nice to have nominal support for that plus SPI flash, at least at the
API level.
K.
On 13/01/17 18:09, Fabio Utzig wrote:
> Now thinking more thoroughly about this I think I arrived at some better
> semantics.
>
> First one correction from what I previously said: since nffs is still
> hardcoded to hal_flash, using only nffs would not require a call do
> disk_register (because no disk_ops is required!).
>
> So, my idea is changing disk_register semantics to be used by "disk"
> drivers instead of by the user app. A disk driver, like flash/mmc/etc,
> would then register itself (like FS's do) with:
>
> disk_register("flash", &flash_ops);
> disk_register("at45db", &at45db_ops);
> disk_register("mmc", &mmc_ops);
>
> The user app will then call "disk_mount" to make this disk available
> like this:
>
> disk_mount("flash0", "nffs", "flash", &flash_device_specific);
> disk_mount("flash1", "nffs", "at45db", &at45db_device_specific);
> disk_mount("mmc0", "fatfs", "mmc", &mmc_device_specific);
>
> Using the fs and disk driver names. This would be a lot more intuitive
> for people with unix background. Also an complimentary "disk_unmount"
> can be added, of course which would be interesting for shell usage with
> devices that support hot disk swap.
>
> What do you think?
Re: Supporting multiple filesystems and "block" devices
Posted by Fabio Utzig <ut...@utzig.org>.
Now thinking more thoroughly about this I think I arrived at some better
semantics.
First one correction from what I previously said: since nffs is still
hardcoded to hal_flash, using only nffs would not require a call do
disk_register (because no disk_ops is required!).
So, my idea is changing disk_register semantics to be used by "disk"
drivers instead of by the user app. A disk driver, like flash/mmc/etc,
would then register itself (like FS's do) with:
disk_register("flash", &flash_ops);
disk_register("at45db", &at45db_ops);
disk_register("mmc", &mmc_ops);
The user app will then call "disk_mount" to make this disk available
like this:
disk_mount("flash0", "nffs", "flash", &flash_device_specific);
disk_mount("flash1", "nffs", "at45db", &at45db_device_specific);
disk_mount("mmc0", "fatfs", "mmc", &mmc_device_specific);
Using the fs and disk driver names. This would be a lot more intuitive
for people with unix background. Also an complimentary "disk_unmount"
can be added, of course which would be interesting for shell usage with
devices that support hot disk swap.
What do you think?
Fabio Utzig
On Fri, Jan 13, 2017, at 08:43 AM, Fabio Utzig wrote:
> Hello,
>
> I've been working on patch that adds support for accessing multiple
> devices and filesystems on Mynewt (overcoming the current single FS and
> "hardcoded" block device per driver). The PR is available here:
>
> https://github.com/apache/incubator-mynewt-core/pull/158
>
> This PR is completely functional with some limitations which I'll talk
> about below and plan to add on a later PR (given the current one is
> already a lot of changes).
>
> The main visible change is that one new package is added which is
> responsible to track what disks use what filesystems. I called it
> "fs/disk". From a user perspective there is an extra step required which
> is a call like this:
>
> disk_register("mmc0", "fatfs", &mmc_ops);
> disk_register("flash0", "nffs", &flash_ops);
>
> This adds a name for the disk, what FS is used (FS's are registered when
> loaded), and what "disk" ops should be used for that named device. The
> disk_ops struct is basically an abstraction of possible operations a
> disk can do like read/write/erase/etc.
>
> All the fs_* functions where updated to support multiple disks. To
> access a file/dir now will required adding the name of the disk as a
> prefix like this:
>
> rc = fs_opendir("mmc0:/", &dir);
> rc = fs_open("flash0:/testfile.txt", FS_ACCESS_READ, &file);
>
> That "mmc0:" and "flash0:" should be the same names that were used when
> registering. Of course, one can always use fatfs_open, nffs_open
> directly and not need to specify the disk. Also for compatibility
> reasons, when only a single FS is used on the project (single FS dep)
> the disk prefix is optional (to not break existing code, tests, etc).
> But the disk_register must be called nonetheless.
>
> What still needs to be done?
>
> 1) Currently FAT is updated to use disk_ops because it only uses
> read/write but it should work on other block devices (would, of course,
> fail on a real flash that requires erase!). NFFS is still calling the
> hal_flash_ functions and should be updated to use the disk_ops interface
> (as soon as it has extra calls for erase, etc).
>
> 2) Also previously discussed in pvt was the fact that we need to pass
> some device context to low-level hal_flash (also mmc_) functions. I
> recently wrote a SPI flash driver for AT45DB which will follow the
> hal_flash_ interface but it needs to receive an extra context to know
> which SPI to use, which configuration, etc. This will probably also add
> another parameter to disk_register with the said context struct.
>
> I'll be fixing the above issues on a new PR.
>
> Please review the current PR, add suggestions, etc.
>
> Cheers,
> Fabio Utzig