You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "Xuanwo (via GitHub)" <gi...@apache.org> on 2023/08/28 06:24:27 UTC
[GitHub] [incubator-opendal] Xuanwo created a discussion: Crazy Ideas: OpenDAL File System
GitHub user Xuanwo created a discussion: Crazy Ideas: OpenDAL File System
Can we build a file system with OpenDAL? Ideally, no. OpenDAL and its underlying storage services do not support random writes, which are crucial for file system use cases. To enable this feature, something like JuiceFS with a metadata service is needed.
However, there is still a need for a file system that:
- Supports both sequential and random reads
- Only supports sequential writes
- Provides other basic file system operations such as listing and stat (which can be easily cached)
There are some projects that fulfill these requirements:
- s3fs-fuse: https://github.com/s3fs-fuse/s3fs-fuse
- mountpoint-s3: https://github.com/awslabs/mountpoint-s3
- gcsfuse: https://github.com/GoogleCloudPlatform/gcsfuse
Perhaps we can implement a fuse via OpenDAL so that users can locally mount storage services and use them as local file systems for specific use cases.
GitHub link: https://github.com/apache/incubator-opendal/discussions/2952
----
This is an automatically sent email for commits@opendal.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@opendal.apache.org
Re: Crazy Ideas: OpenDAL File System
Posted by "Barre (via GitHub)" <gi...@apache.org>.
GitHub user Barre added a comment to the discussion: Crazy Ideas: OpenDAL File System
I think https://github.com/Barre/ZeroFS kind of fits that description :)
I am in the process of integrating OpenDAL.
GitHub link: https://github.com/apache/opendal/discussions/2952#discussioncomment-13913777
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
[GitHub] [incubator-opendal] oowl added a comment to the discussion: Crazy Ideas: OpenDAL File System
Posted by "oowl (via GitHub)" <gi...@apache.org>.
GitHub user oowl added a comment to the discussion: Crazy Ideas: OpenDAL File System
I'm having a bit of trouble grasping this concept. openDAL provides a set of library APIs for users. Therefore, we suggest users should utilize some bindings or the core rust API for their purpose. However, the term 'ofs' seems perplexing to me. Could it be more akin to the 'oay' frontend service?
GitHub link: https://github.com/apache/incubator-opendal/discussions/2952#discussioncomment-6856143
----
This is an automatically sent email for commits@opendal.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@opendal.apache.org
[GitHub] [incubator-opendal] Xuanwo added a comment to the discussion: Crazy Ideas: OpenDAL File System
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: Crazy Ideas: OpenDAL File System
I have been researching about virtio/virtiofs/vDPA/VDUSE recently. Maybe we can provide both virtio and fuse support for `ofs`.
GitHub link: https://github.com/apache/incubator-opendal/discussions/2952#discussioncomment-6838956
----
This is an automatically sent email for commits@opendal.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@opendal.apache.org
[GitHub] [incubator-opendal] Xuanwo edited a discussion: Crazy Ideas: OpenDAL File System
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo edited a discussion: Crazy Ideas: OpenDAL File System
Can we build a file system with OpenDAL? Ideally, no. OpenDAL and its underlying storage services do not support random writes, which are crucial for file system use cases. To enable this feature, something like JuiceFS with a metadata service is needed.
However, there is still a need for a file system that:
- Supports both sequential and random reads
- Only supports sequential writes
- Provides other basic file system operations such as listing and stat (which can be easily cached)
There are some projects that fulfill these requirements:
- s3fs-fuse: https://github.com/s3fs-fuse/s3fs-fuse
- mountpoint-s3: https://github.com/awslabs/mountpoint-s3
- gcsfuse: https://github.com/GoogleCloudPlatform/gcsfuse
Perhaps we can implement a fuse via OpenDAL so that users can locally mount storage services and use them as local file systems for specific use cases.
Similiar ideas
- https://github.com/apache/incubator-opendal/issues/2625
GitHub link: https://github.com/apache/incubator-opendal/discussions/2952
----
This is an automatically sent email for commits@opendal.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@opendal.apache.org
[GitHub] [incubator-opendal] Xuanwo added a comment to the discussion: Crazy Ideas: OpenDAL File System
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: Crazy Ideas: OpenDAL File System
I'm also thinking of splitting oay into different binaries to make it easier to use.
GitHub link: https://github.com/apache/incubator-opendal/discussions/2952#discussioncomment-6859453
----
This is an automatically sent email for commits@opendal.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@opendal.apache.org
[GitHub] [incubator-opendal] Xuanwo added a comment to the discussion: Crazy Ideas: OpenDAL File System
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: Crazy Ideas: OpenDAL File System
Yes. More like part of oay.
GitHub link: https://github.com/apache/incubator-opendal/discussions/2952#discussioncomment-6859445
----
This is an automatically sent email for commits@opendal.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@opendal.apache.org
[GitHub] [incubator-opendal] Xuanwo edited a comment on the discussion: Crazy Ideas: OpenDAL File System
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo edited a comment on the discussion: Crazy Ideas: OpenDAL File System
I have been researching about virtio/virtiofs/vDPA/VDUSE recently. Maybe we can provide both virtio and fuse support for `ofs`?
## For virtio
Users can mount ofs to container or vm directly. All control message send via a char device and all data send via `virtio` which is much faster than `fuse`.
This solution is linux only.
## For fuse
Users can mount ofs to host directly via fuse. All control message and data are sent via `fuse` device.
Nearly all platform supported.
---
We can build `ofs-core` and provide `ofs-virtiofs` and `ofs-fuse` for users.
GitHub link: https://github.com/apache/incubator-opendal/discussions/2952#discussioncomment-6838956
----
This is an automatically sent email for commits@opendal.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@opendal.apache.org
[GitHub] [incubator-opendal] Xuanwo added a comment to the discussion: Crazy Ideas: OpenDAL File System
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: Crazy Ideas: OpenDAL File System
A PoC written by @Inokinoki
https://github.com/Inokinoki/DalFs
GitHub link: https://github.com/apache/incubator-opendal/discussions/2952#discussioncomment-6956606
----
This is an automatically sent email for commits@opendal.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@opendal.apache.org