You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "mangelajo (via GitHub)" <gi...@apache.org> on 2025/03/05 15:42:17 UTC
[I] new feature: Support OCI artifacts [opendal]
mangelajo opened a new issue, #5695:
URL: https://github.com/apache/opendal/issues/5695
### Feature Description
I would love to see support in opendal for OCI artifact reading/extraction (see [1])
[1] https://oras.land/docs/quickstart
### Problem and Solution
We have integrated support in jumpstarter for opendal as our storage layer for accessing storage, or passing presigned storage access to edge elements.
https://github.com/jumpstarter-dev/jumpstarter/tree/main/packages/jumpstarter-driver-opendal/jumpstarter_driver_opendal
handling OCI artifacts is becoming standard in the industry to transport device images, or firmware, and we would like to take advantage of that.
CC: @NickCao
### Additional Context
We don't have capacity to work on this at the moment.
### Are you willing to contribute to the development of this feature?
- [ ] Yes, I am willing to contribute to the development of this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support OCI artifacts [opendal]
Posted by "erickguan (via GitHub)" <gi...@apache.org>.
erickguan commented on issue #5695:
URL: https://github.com/apache/opendal/issues/5695#issuecomment-2932355313
Looks neat! Thanks for the idea.
How do you represent local registry (loaded in docker)?
What about metadata access? They are two kinds:
1. Attestations, mostly a large JSON
2. Annotations, e.g. manifests such as multi-platform index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support OCI artifacts [opendal]
Posted by "DCjanus (via GitHub)" <gi...@apache.org>.
DCjanus commented on issue #5695:
URL: https://github.com/apache/opendal/issues/5695#issuecomment-4589296897
I’m interested in trying a basic PR for this, mostly to get things moving and to expose more concrete design questions during implementation.
Before starting, I want to share my current understanding of a possible initial shape. This may differ from what others have in mind, so please treat it as a starting point. I’m happy to adjust if there are better suggestions.
## Proposed initial scope
I’m currently leaning toward starting with **local OCI image layout** first, instead of supporting remote registries or the local Docker daemon directly. This should let us focus on the OpenDAL path model, root handling, layer merging, whiteout behavior, and `read` / `stat` / `list` semantics.
The first version could support configuration like this:
| Option | Example | Notes |
| --- | --- | --- |
| `layout` | `/path/to/oci-layout` | Local OCI image layout directory |
| `reference` | `latest` | Required, used to select a manifest from `index.json` |
| `platform` | `linux/amd64` | Required, to avoid implicit image selection |
| `root` | `/` | Optional, access boundary inside the image rootfs |
Then OpenDAL paths would be relative to the configured root inside the resolved, merged filesystem. For example, with `root = "/"`:
```rust
op.read("etc/os-release").await?; // reads /etc/os-release from the image rootfs
op.list("usr/bin/").await?;
```
With this model, `list("")` or `list("/")` would list the configured root inside the image filesystem, not registry tags, image references, or blob paths inside the OCI layout. Remote registries, local Docker / Podman / containerd storage, archives, per-layer access, attestations, annotations, and SBOMs can be left out of the first PR.
<details>
<summary>Open questions</summary>
I have a few uncertain points that I’d like to check before starting.
### Default root
My current assumption is that the default `root` should be the image rootfs `/`, not the image `WORKDIR`, because OpenDAL’s `root` usually represents a storage access boundary rather than a process execution context. But I’m not sure whether others would expect different behavior here.
### URI support
I’m not planning to implement `Operator::from_uri` support in the first PR because I don’t yet have a clear URI design in mind.
I like the transport-reference style mentioned by @NickCao, but I’m still unsure how that model should map into an OpenDAL URI while avoiding nested schemes like `container://docker://...`. So I’d prefer to start with builder-based configuration first and revisit URI support once the model is clearer.
Since this may turn out to be a larger design space than it first appears, I may mark the initial implementation as experimental if that seems acceptable, so we have room to adjust the config, URI shape, or metadata exposure based on what we learn during implementation.
</details>
Does this sound like a reasonable direction for a first small PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support OCI artifacts [opendal]
Posted by "NickCao (via GitHub)" <gi...@apache.org>.
NickCao commented on issue #5695:
URL: https://github.com/apache/opendal/issues/5695#issuecomment-2701653839
One problem I can see with oci artifacts is that each artifact itself is likely a directory of files (encapsulated as a tarball), so we really have two layers of "directories" here, the oci registry namespace/image/tag triplet, and the directory tree within a single artifact. For starters we might focus on the operations within a single artifact, yet you cannot implement efficient list operation on tarballs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support OCI artifacts [opendal]
Posted by "NickCao (via GitHub)" <gi...@apache.org>.
NickCao commented on issue #5695:
URL: https://github.com/apache/opendal/issues/5695#issuecomment-2932390684
> thoughts on this path model?
One suggestion: instead of using the `oci://` schema, use `container://`, followed by a container transport reference as defined in https://github.com/containers/image/blob/main/docs/containers-transports.5.md
And: would this create ambiguity if ref is omitted (implying latest), or with multi level repo names (e.g `example.com/foo/bar/baz/image`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support OCI artifacts [opendal]
Posted by "erickguan (via GitHub)" <gi...@apache.org>.
erickguan commented on issue #5695:
URL: https://github.com/apache/opendal/issues/5695#issuecomment-4598511977
@DCjanus Go ahead!
There are two paths for your development:
1. fork and use a branch: easy to get started and you could get some feedback from CI.
2. build as a separate service and integrate later: 100% under your control and you can iterate until we feel ready.
For your open questions:
> My current assumption is that the default root should be the image rootfs /, not the image WORKDIR, because OpenDAL’s root usually represents a storage access boundary rather than a process execution context.
Reasonable to start with /.
> I’m not planning to implement Operator::from_uri support in the first PR because I don’t yet have a clear URI design in mind.
If you start with 2 development approach, this is trivial to manage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support OCI artifacts [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #5695:
URL: https://github.com/apache/opendal/issues/5695#issuecomment-2701558050
Thank you @mangelajo for this issue. I'm willing help make some progress here. First of all, I need to learn how we can map OCI to Opendal's calling convention.
OpenDAL creates a model that allows us to:
- Read data from `path`
- Write data to `path`
- List files from `path`
To build a useful service, we should at least support `read` and `list`. Can we implement this over OCI artifacts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support OCI artifacts [opendal]
Posted by "erickguan (via GitHub)" <gi...@apache.org>.
erickguan commented on issue #5695:
URL: https://github.com/apache/opendal/issues/5695#issuecomment-2703693094
This also helps tooling. My `dive` doesn't work because of a bug.
Do you strictly want the OCI image layout tarball? Registries, although similar and related, are not the same strictly. If it's only the image tarball, do you want to include estargz too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support OCI artifacts [opendal]
Posted by "PSU3D0 (via GitHub)" <gi...@apache.org>.
PSU3D0 commented on issue #5695:
URL: https://github.com/apache/opendal/issues/5695#issuecomment-2925495554
Any movement on this?
@NickCao thoughts on this path model?
```
oci://{registry}/{repo}/@{ref}/{inner?}
registry = ghcr.io | registry-1.docker.io | ...
repo = namespace/name
ref = <tag> | sha256:<digest> (OCI "reference")
inner = optional path *inside* the unpacked rootfs
```
Registry + repo acts like the bucket field in S3.
We reuse the first path segment for the OCI reference (tag / digest)
A plain list("/") therefore returns tags and digests - matches what most users expect when they "list a repository."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support OCI artifacts [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #5695:
URL: https://github.com/apache/opendal/issues/5695#issuecomment-4616031856
This issue goes into two different directions: one is for accessing container images and the other one is to access the rootfs inside an image.
For the second one: the target is only support local images? And we can read from the registry?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org