You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opendal.apache.org by "clbarnes (via GitHub)" <gi...@apache.org> on 2026/01/14 10:22:41 UTC
Guidance on zip-like stores
GitHub user clbarnes created a discussion: Guidance on zip-like stores
I'm interested in making [fjall-rs/sfa](github.com/fjall-rs/sfa) accessible through opendal. It's conceptually similar to a zip file but less complicated - the "file entries" (sections) are basically concatenated, uncompressed, and then there's a footer with their keys along with an offset and length. I may also be interested in zip files themselves.
I'd like to be able to read these remotely, e.g. an SFA file on S3. I think that means I should be working on a Layer rather than an Operator, although that might break some assumptions about getting AccessorInfo from the Layer.
I'd intend initially to make this accessor read-only and store the index in memory, probably on first read (so put it behind an `RwLock<Option<...>>`).
Does this sound like the right track?
GitHub link: https://github.com/apache/opendal/discussions/7148
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Guidance on zip-like stores
Posted by "erickguan (via GitHub)" <gi...@apache.org>.
GitHub user erickguan added a comment to the discussion: Guidance on zip-like stores
Ok. Xuanwo's idea is the best approach IMO. The URI might require some work but it would be quite straightforward setting up a one level zip like operator.
GitHub link: https://github.com/apache/opendal/discussions/7148#discussioncomment-17088595
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Guidance on zip-like stores
Posted by "erickguan (via GitHub)" <gi...@apache.org>.
GitHub user erickguan added a comment to the discussion: Guidance on zip-like stores
Does different zip format support seek? Container folks spent loads of effort supporting a lazy loadable image format. I am kind of worrying about the effort to the return.
Another idea is to have some sort of compression layer. Then you would use object store as the zip structure.
GitHub link: https://github.com/apache/opendal/discussions/7148#discussioncomment-17078163
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Guidance on zip-like stores
Posted by "luanmm (via GitHub)" <gi...@apache.org>.
GitHub user luanmm added a comment to the discussion: Guidance on zip-like stores
I found OpenDAL recently when I was working with ZIP files, in an efficient way (from a streaming/partial reading perspective, not full decompression), and it was very adherent to my use case (as the original library I was working on was doing HTTP buffered readings and other strategies that OpenDAL already considers).
So I thought OpenDAL could make a move towards this "archive" scenario, as it already has very nice examples of opening files from multiple protocols and serving as an object store, for instance... or even the "ofs" project. In that way, archive files could be read, generally, as "virtual file systems" somehow. When I open a ZIP file, what I'm doing, in practice, is listing files like I would do when I open an S3 bucket. Even Windows (which I haven't touched for a while, by the way) has some kind of "folder-like" integration in its iconic "file explorer".
In terms of API, a ZIP file could be "opened" in OpenDAL as any other operator. The particular case here is that it can be thought of as "an operator inside another operator", as I may have ZIP files inside S3 buckets, or served directly via HTTP or even the filesystem. This is the only "challenge" that I see in terms of how OpenDAL could handle this kind of scenario.
Example of how this could be represented as URIs:
```bash
# 1) as a "path", directly (should be parsed considering the extension, which is not great)
s3://<bucket>/folder-a/file.zip/inner-file-1.txt
# 2) same as above, but with a "hint" from protocol, as used in Git and others
s3+zip://<bucket>/folder-a/file.zip/inner-file-1.txt
```
There are other more complex scenarios, like nested "VFS":
```bash
s3://<bucket>/data.iso/file-a.zip/file-b.zip/inner-file-a.7z/target-file.txt
# considering "hints" in protocol (yeah, just to be a little bit crazy here)
s3+iso+zip+7z://<bucket>/data.iso/file-a.zip/file-b.zip/inner-file-a.7z/target-file.txt
```
This is taking it to the limit and considering some edge cases (which aren't impossible anyway).
Thinking from an architectural perspective, it seems to me that this would make some kind of "processor" entity emerge, that could be enabled when needed, in options/protocol-hints or as a layer (e.g. a middleware that can intercept processing, taking only the part of the URI that it has interest in, like `.zip`, then passes the operator that handles ZIP-like VFS to the next one, etc).
Well... this is just a big "brainstorm" here, just to say that I would think that OpenDAL would be even more incredible supporting this kind of scenario, as it just rounds out what it currently supports.
GitHub link: https://github.com/apache/opendal/discussions/7148#discussioncomment-16931651
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Guidance on zip-like stores
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: Guidance on zip-like stores
It feels more natural to me to design SFA so that the library itself can accept an OpenDAL operator (rather than a Path). This way, all users of SFA can read directly from S3. Is `fjall-rs` open to accept a change to sfa or building a `sfa-opendal`?
GitHub link: https://github.com/apache/opendal/discussions/7148#discussioncomment-15494623
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Guidance on zip-like stores
Posted by "clbarnes (via GitHub)" <gi...@apache.org>.
GitHub user clbarnes added a comment to the discussion: Guidance on zip-like stores
I've already implemented this (but not PR'd it yet) - it's quite a lot simpler than SFA as an Accessor. My thinking was that it would be nice for people to be able to use it as a drop-in replacement for whatever other store they're using.
My use case is [Zarr](https://zarr.dev/), a scheme for storing and accessing ND arrays in chunks. The rust implementation already has an opendal adapter. There is interest in a single-file format for portability so I thought it would be better to contribute an SFA accessor and hook into existing systems rather than having to write separate bridges for SFA, zip etc..
GitHub link: https://github.com/apache/opendal/discussions/7148#discussioncomment-15494779
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Guidance on zip-like stores
Posted by "clbarnes (via GitHub)" <gi...@apache.org>.
GitHub user clbarnes edited a discussion: Guidance on zip-like stores
I'm interested in making [fjall-rs/sfa](github.com/fjall-rs/sfa) accessible through opendal. It's conceptually similar to a zip file but less complicated - the "file entries" (sections) are basically concatenated, uncompressed, and then there's a footer with their keys along with an offset and length. I may also be interested in zip files themselves.
I'd like to be able to read these remotely, e.g. an SFA file on S3. I think that means I should be working on a Layer rather than an Operator, although that might break some assumptions about getting AccessorInfo from the Layer.
I'd intend initially to make this accessor read-only and store the index in memory, probably on first read (so put it behind an `RwLock<Option<...>>`).
Does this sound like the right track?
---
**Mailing list thread:** https://lists.apache.org/thread/4547mv3g2bgp2dv7ykh4zf437skmhwr7
GitHub link: https://github.com/apache/opendal/discussions/7148
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Guidance on zip-like stores
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: Guidance on zip-like stores
It seems to be an interesting idea.
I think it's possible to build a services like `opendal-service-zip`, `opendal-services-7z`, at least for read-only. And I think not much people want to use opendal to really write them.
Those service should accept an existing `Operator` inside, so users can pass whatever storage they want to read.
And from the uri level, we might be able to do this:
```
zip://path-to-file?source=s3://bucket/xxx.zip
```
GitHub link: https://github.com/apache/opendal/discussions/7148#discussioncomment-17014373
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Guidance on zip-like stores
Posted by "clbarnes (via GitHub)" <gi...@apache.org>.
GitHub user clbarnes added a comment to the discussion: Guidance on zip-like stores
The file entries inside zip archives support seek so long as the entries are not compressed.
GitHub link: https://github.com/apache/opendal/discussions/7148#discussioncomment-17086865
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Guidance on zip-like stores
Posted by "clbarnes (via GitHub)" <gi...@apache.org>.
GitHub user clbarnes added a comment to the discussion: Guidance on zip-like stores
We've been struggling with stores-in-stores over at [zarr](https://zarr.dev/). Some proposals include
- a new specification as a superset of URLs https://github.com/jbms/url-pipeline, like `s3://bucket/path/to/archive.zip|zip:path/within/zip.zarr/`. The "related work" section points to:
- fsspec, similar to the above, but using `::` as a delimiter
- apache commons VFS, which puts all the schemes first and then all the non-scheme parts separated by `!`
- java jar URLs, like apache commons but specific to jars
- gdal VFS, nesting paths in `{}`
- GNOME VFS URL-encodes the outer path as the host of a URL
GitHub link: https://github.com/apache/opendal/discussions/7148#discussioncomment-17015936
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org