You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nuttx.apache.org by Dmitri Shilov <ds...@cthru.xyz> on 2024/03/22 17:26:37 UTC
exFAT or other large options for file system storage.
Good day,
I am looking for a way to store upwards of 100's GB to SDXC in a continuously recording embedded project. I've found that exFAT was made available royalty-free for Linux 5.4 onwards, but nothing on embedded systems. Would I have to work through one of Microsoft's implementation partners to get exFAT support for our project? I found a mention of exFAT support with NuttX eVFAT in a Sony presentation back from 2016, but the Spresense SDK doesn't support exFAT either.
Are there any other options within NuttX to store more than fat32 limits? Do I need to multiplex multiple memory devices?
Any advice would be appreciated.
Regards,
Dmitri Shilov.
exFAT or other large options for file system storage.
Posted by Dmitri Shilov <ds...@cthru.xyz>.
Thank you for the guidance. Regarding exFAT, I've reached out to its licencing body to see what licence is necessary for commercial use in non-resale items. Everything I've read so far indicates that even with available implementations, if I build it into my product I have to comply with exFAT licensing restrictions, so even if Apache has licence to use it as part of OIN, that right would not be transferred to products built with NuttX by others.
Xiao, FatFS looks most appealing as far as implementations go.
Regards,
Dmitri Shilov.
-------- Original Message --------
On 2024-03-22 9:58 p.m., Xiang Xiao <xi...@gmail.com> wrote:
> If you search exfat implementation: FatFS from
> http://elm-chan.org/fsw/ff/00index_e.html is a good choice, since it's:
>
> 1. has a better friend license than GPL/LGPL
> 2. designed for small embedded device natively
>
> BTW, if the community wants another fat implementation, we could upstream
> NuttX FS wrapper, which has been used in the product for several years.
>
> On Sat, Mar 23, 2024 at 3:51 AM Tomek CEDRO <to...@cedro.info> wrote:
>
> > On Fri, Mar 22, 2024 at 7:54 PM Alan C. Assis <ac...@gmail.com> wrote:
> > > Hi Dmitri,
> > > But I think even an open-source implementation will end-up paying
> > > royalties, unless you are member of OIN:
> > > https://www.paragon-software.com/exfat-license/
> >
> > Here is a good page on exFAT along with Legal Status and List of
> > Implementations sections:
> >
> > https://en.wikipedia.org/wiki/ExFAT
> >
> > Looks like Paragon Software is a proprietary implementation too.
> >
> >
> > OIN membership is something else and it seems free:
> >
> > https://openinventionnetwork.com/our-members/funding-members-and-licensees/
> >
> > According to wikipedia Apache is covered by OIN, not sure if this is
> > about web server or all of the Apache Foundation projects? Maybe this
> > is the way to get exFAT in NuttX?
> >
> > https://en.wikipedia.org/wiki/Open_Invention_Network
> >
> >
> > > So, maybe you could think about some alternatives, like porting EXT3 to
> > > NuttX (but that is not an easy task, since even FreeBSD doesn't support
> > > EXT3).
> >
> > On FreeBSD we have lots of filesystems supported by FUSE and it works
> > pretty well now. Its not a part of the system but user can install it
> > from ports/packages. This is the implementation in use (GNU license):
> >
> > https://github.com/relan/exfat
> >
> > Here is a detailed filesystem comparison page:
> >
> > https://en.wikipedia.org/wiki/Comparison_of_file_systems
> >
> > Before ZFS was introduced to FreeBSD we used UFS(2) (Unix File System)
> >
> > https://en.wikipedia.org/wiki/Unix_File_System
> >
> >
> > The most universal and open filesystem is UDF (POSIX compliant) but it
> > varies in implementations and versions and no write support for all of
> > the versions so its kinda messy and platform specific. It was (and
> > probably still is) not really a good multiplatform replacement. I
> > tried using it long time ago with FreeBSD, Linux and Windows as common
> > filesystem on data partition for all of the OS on a single machine.
> > But I sticked to EXT2. Now I simply use ZFS on FreeBSD. But ZFS even
> > nowadays is still not a solution to work with a single data partition
> > on various OS and definitely not an embedded solution. People simply
> > used separate machine with NAS for storage over NFS. Idea for
> > universal filesystem seems dead :-(
> >
> > UDF is widely used nowadays on top of ISO9660 to burn optical disks
> > with large files (i.e. double layer DVD or BluRay). All systems can
> > read it. Not all can write it on the fly (it is usually used to create
> > static ISO+UDF image). But in theory it seems better than exFAT!
> >
> > https://en.wikipedia.org/wiki/Universal_Disk_Format
> >
> > I found some old GSoC 2012 page on UDF implementation in FreeBSD
> > (based on NetBSD code):
> >
> > https://wiki.freebsd.org/SummerOfCode2012/UDFImplementation
> >
> >
> > Having a modern universal filesystem support in NuttX seems important.
> >
> > Maybe UDF 2.60 implementation could happen?
> >
> > exFAT seems more popular. Maybe Apache could help here over the OIN?
> > And most of the existing exFAT implementations are covered by GNU
> > license so that may be no go for some companies / projects.
> >
> > And the standard question who is willing to invest time / cash for
> > that implementation?
> >
> >
> > UDF has some interesting features that may beneficial for instance
> > when streaming video of backing up data to a memory for instance VAT /
> > Virtual Allocation Table mode that is incremental write once, or
> > Spared mode that is used to write to rewritable media. And there is a
> > BSD licensed implementation already out there.
> >
> > Also when recording multimedia stream is it VERY important to choose
> > proper container format. For instance MP4 must be properly closed in
> > order to be opened - kind of nightmare of aviation recording when
> > camera hangs/breaks in the middle of flight and then folks cannot even
> > open a recording. MKV seems better choice. But (in)famous GoPro still
> > uses MP4 ;-)
> >
> > --
> > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >
>
Re: exFAT or other large options for file system storage.
Posted by Xiang Xiao <xi...@gmail.com>.
If you search exfat implementation: FatFS from
http://elm-chan.org/fsw/ff/00index_e.html is a good choice, since it's:
1. has a better friend license than GPL/LGPL
2. designed for small embedded device natively
BTW, if the community wants another fat implementation, we could upstream
NuttX FS wrapper, which has been used in the product for several years.
On Sat, Mar 23, 2024 at 3:51 AM Tomek CEDRO <to...@cedro.info> wrote:
> On Fri, Mar 22, 2024 at 7:54 PM Alan C. Assis <ac...@gmail.com> wrote:
> > Hi Dmitri,
> > But I think even an open-source implementation will end-up paying
> > royalties, unless you are member of OIN:
> > https://www.paragon-software.com/exfat-license/
>
> Here is a good page on exFAT along with Legal Status and List of
> Implementations sections:
>
> https://en.wikipedia.org/wiki/ExFAT
>
> Looks like Paragon Software is a proprietary implementation too.
>
>
> OIN membership is something else and it seems free:
>
> https://openinventionnetwork.com/our-members/funding-members-and-licensees/
>
> According to wikipedia Apache is covered by OIN, not sure if this is
> about web server or all of the Apache Foundation projects? Maybe this
> is the way to get exFAT in NuttX?
>
> https://en.wikipedia.org/wiki/Open_Invention_Network
>
>
> > So, maybe you could think about some alternatives, like porting EXT3 to
> > NuttX (but that is not an easy task, since even FreeBSD doesn't support
> > EXT3).
>
> On FreeBSD we have lots of filesystems supported by FUSE and it works
> pretty well now. Its not a part of the system but user can install it
> from ports/packages. This is the implementation in use (GNU license):
>
> https://github.com/relan/exfat
>
> Here is a detailed filesystem comparison page:
>
> https://en.wikipedia.org/wiki/Comparison_of_file_systems
>
> Before ZFS was introduced to FreeBSD we used UFS(2) (Unix File System)
>
> https://en.wikipedia.org/wiki/Unix_File_System
>
>
> The most universal and open filesystem is UDF (POSIX compliant) but it
> varies in implementations and versions and no write support for all of
> the versions so its kinda messy and platform specific. It was (and
> probably still is) not really a good multiplatform replacement. I
> tried using it long time ago with FreeBSD, Linux and Windows as common
> filesystem on data partition for all of the OS on a single machine.
> But I sticked to EXT2. Now I simply use ZFS on FreeBSD. But ZFS even
> nowadays is still not a solution to work with a single data partition
> on various OS and definitely not an embedded solution. People simply
> used separate machine with NAS for storage over NFS. Idea for
> universal filesystem seems dead :-(
>
> UDF is widely used nowadays on top of ISO9660 to burn optical disks
> with large files (i.e. double layer DVD or BluRay). All systems can
> read it. Not all can write it on the fly (it is usually used to create
> static ISO+UDF image). But in theory it seems better than exFAT!
>
> https://en.wikipedia.org/wiki/Universal_Disk_Format
>
> I found some old GSoC 2012 page on UDF implementation in FreeBSD
> (based on NetBSD code):
>
> https://wiki.freebsd.org/SummerOfCode2012/UDFImplementation
>
>
> Having a modern universal filesystem support in NuttX seems important.
>
> Maybe UDF 2.60 implementation could happen?
>
> exFAT seems more popular. Maybe Apache could help here over the OIN?
> And most of the existing exFAT implementations are covered by GNU
> license so that may be no go for some companies / projects.
>
> And the standard question who is willing to invest time / cash for
> that implementation?
>
>
> UDF has some interesting features that may beneficial for instance
> when streaming video of backing up data to a memory for instance VAT /
> Virtual Allocation Table mode that is incremental write once, or
> Spared mode that is used to write to rewritable media. And there is a
> BSD licensed implementation already out there.
>
> Also when recording multimedia stream is it VERY important to choose
> proper container format. For instance MP4 must be properly closed in
> order to be opened - kind of nightmare of aviation recording when
> camera hangs/breaks in the middle of flight and then folks cannot even
> open a recording. MKV seems better choice. But (in)famous GoPro still
> uses MP4 ;-)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>
Re: exFAT or other large options for file system storage.
Posted by Tomek CEDRO <to...@cedro.info>.
On Fri, Mar 22, 2024 at 7:54 PM Alan C. Assis <ac...@gmail.com> wrote:
> Hi Dmitri,
> But I think even an open-source implementation will end-up paying
> royalties, unless you are member of OIN:
> https://www.paragon-software.com/exfat-license/
Here is a good page on exFAT along with Legal Status and List of
Implementations sections:
https://en.wikipedia.org/wiki/ExFAT
Looks like Paragon Software is a proprietary implementation too.
OIN membership is something else and it seems free:
https://openinventionnetwork.com/our-members/funding-members-and-licensees/
According to wikipedia Apache is covered by OIN, not sure if this is
about web server or all of the Apache Foundation projects? Maybe this
is the way to get exFAT in NuttX?
https://en.wikipedia.org/wiki/Open_Invention_Network
> So, maybe you could think about some alternatives, like porting EXT3 to
> NuttX (but that is not an easy task, since even FreeBSD doesn't support
> EXT3).
On FreeBSD we have lots of filesystems supported by FUSE and it works
pretty well now. Its not a part of the system but user can install it
from ports/packages. This is the implementation in use (GNU license):
https://github.com/relan/exfat
Here is a detailed filesystem comparison page:
https://en.wikipedia.org/wiki/Comparison_of_file_systems
Before ZFS was introduced to FreeBSD we used UFS(2) (Unix File System)
https://en.wikipedia.org/wiki/Unix_File_System
The most universal and open filesystem is UDF (POSIX compliant) but it
varies in implementations and versions and no write support for all of
the versions so its kinda messy and platform specific. It was (and
probably still is) not really a good multiplatform replacement. I
tried using it long time ago with FreeBSD, Linux and Windows as common
filesystem on data partition for all of the OS on a single machine.
But I sticked to EXT2. Now I simply use ZFS on FreeBSD. But ZFS even
nowadays is still not a solution to work with a single data partition
on various OS and definitely not an embedded solution. People simply
used separate machine with NAS for storage over NFS. Idea for
universal filesystem seems dead :-(
UDF is widely used nowadays on top of ISO9660 to burn optical disks
with large files (i.e. double layer DVD or BluRay). All systems can
read it. Not all can write it on the fly (it is usually used to create
static ISO+UDF image). But in theory it seems better than exFAT!
https://en.wikipedia.org/wiki/Universal_Disk_Format
I found some old GSoC 2012 page on UDF implementation in FreeBSD
(based on NetBSD code):
https://wiki.freebsd.org/SummerOfCode2012/UDFImplementation
Having a modern universal filesystem support in NuttX seems important.
Maybe UDF 2.60 implementation could happen?
exFAT seems more popular. Maybe Apache could help here over the OIN?
And most of the existing exFAT implementations are covered by GNU
license so that may be no go for some companies / projects.
And the standard question who is willing to invest time / cash for
that implementation?
UDF has some interesting features that may beneficial for instance
when streaming video of backing up data to a memory for instance VAT /
Virtual Allocation Table mode that is incremental write once, or
Spared mode that is used to write to rewritable media. And there is a
BSD licensed implementation already out there.
Also when recording multimedia stream is it VERY important to choose
proper container format. For instance MP4 must be properly closed in
order to be opened - kind of nightmare of aviation recording when
camera hangs/breaks in the middle of flight and then folks cannot even
open a recording. MKV seems better choice. But (in)famous GoPro still
uses MP4 ;-)
--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
Re: exFAT or other large options for file system storage.
Posted by "Alan C. Assis" <ac...@gmail.com>.
Hi Dmitri,
I think exFAT is used in the Sony audio products (not in Spresense SDK as
you confirmed) and the only place I saw about eVFAT was in Masayuki's
presentation.
Yes, getting some implementation from some Microsoft's partners is a
possible way, or you can hire someone (or some company) to implement it,
since the specification is open:
https://learn.microsoft.com/en-us/windows/win32/fileio/exfat-specification
But I think even an open-source implementation will end-up paying
royalties, unless you are member of OIN:
https://www.paragon-software.com/exfat-license/
There is also an old thread from 2009 about it:
https://www.reddit.com/r/linux/comments/adcur/microsoft_licenses_out_exfat_file_system_in_the/
So, maybe you could think about some alternatives, like porting EXT3 to
NuttX (but that is not an easy task, since even FreeBSD doesn't support
EXT3).
Maybe an even simpler solution is dividing your storage data in files lower
than 4GB to let you use FAT32 SDXC cards. You could ship a software can
read these cards and concatenate these files together.
Best Regards,
Alan
On Fri, Mar 22, 2024 at 2:27 PM Dmitri Shilov <ds...@cthru.xyz> wrote:
> Good day,
>
> I am looking for a way to store upwards of 100's GB to SDXC in a
> continuously recording embedded project. I've found that exFAT was made
> available royalty-free for Linux 5.4 onwards, but nothing on embedded
> systems. Would I have to work through one of Microsoft's implementation
> partners to get exFAT support for our project? I found a mention of exFAT
> support with NuttX eVFAT in a Sony presentation back from 2016, but the
> Spresense SDK doesn't support exFAT either.
>
> Are there any other options within NuttX to store more than fat32 limits?
> Do I need to multiplex multiple memory devices?
>
> Any advice would be appreciated.
>
> Regards,
> Dmitri Shilov.
>