You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opendal.apache.org by "Xuanwo (via GitHub)" <gi...@apache.org> on 2024/11/05 13:58:53 UTC
discuss: Remove Metakey Concept
GitHub user Xuanwo created a discussion: discuss: Remove Metakey Concept
[`Metakey`](https://docs.rs/opendal/latest/opendal/enum.Metakey.html) is designed for users to query metadata from storage services. However, it is so complex that users struggle to use it correctly, while developers find it challenging to implement properly.
When I designed `Metakey`, I expect users to use them like the following:
```rust
let lister = op.lister_with(path).metakey(Metakey::ContentLength | Metakey::ContentType).await?;
```
Users can select the `meta` they want, and opendal will determine whether to fetch more as needed.
However, this mechanism has been used incorrectly in many ways.
## Extra stat call
Users aren't aware that setting `Metakey::Full` results in an additional stat call for each entry, which can make list performance up to 1000 times slower.
## Unexpected metakey used
Many users will use `Metakey` in this way:
```rust
let entries = op.lister_with(path).metakey(Metakey::ContentLength).await?.collect().await;
...
entry.metadata().content_type(); // This meta is not queried.
```
## Overlap with `version`
Lister now includes a new argument called `version(bool)`, which determines whether to list versioned objects. This somewhat overlaps with `Metakey::Version`, potentially causing confusion.
---
Although I still love `Metakey`, it is complex and can be easily misused in the wrong API, so it should not exist in OpenDAL.
I suggest to remove this concept from the OpenDAL, and with the following changes:
- Add `MetadataCapability` which a struct like:
```rust
struct MetadataCapability {
content_length: bool,
content_type: bool,
...
}
```
Services can return `MetadataCapability` to indicate how much metadata will be included in the returned data. Users can use this capability to decide whether to perform extra stat or not.
- Don't perform `stat` during `list`
The list will no longer perform `stat` operations. Users who want this functionality can easily use the `Stream` API to compose them instead.
After those changes, OpenDAL's `Lister` will be much simpler and more predictable.
What do you think? I will propose an RFC if no objection.
GitHub link: https://github.com/apache/opendal/discussions/5287
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: discuss: Remove Metakey Concept
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo edited a comment on the discussion: discuss: Remove Metakey Concept
Thank you @erickguan and @alexwilcoxson-rel for the feedback. I'm preparing an RFC for this change.
---
Proposed: https://github.com/apache/opendal/pull/5313
GitHub link: https://github.com/apache/opendal/discussions/5287#discussioncomment-11227349
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: discuss: Remove Metakey Concept
Posted by "alexwilcoxson-rel (via GitHub)" <gi...@apache.org>.
GitHub user alexwilcoxson-rel added a comment to the discussion: discuss: Remove Metakey Concept
I just ran into this with object_store integration stat'ing a ton of leftover files in a deltalake log as Version is not a property on ADLS, so +1!
GitHub link: https://github.com/apache/opendal/discussions/5287#discussioncomment-11191961
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: discuss: Remove Metakey Concept
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: discuss: Remove Metakey Concept
Thank you @erickguan and @alexwilcoxson-rel for the feedback. I'm preparing an RFC for this change.
GitHub link: https://github.com/apache/opendal/discussions/5287#discussioncomment-11227349
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: discuss: Remove Metakey Concept
Posted by "erickguan (via GitHub)" <gi...@apache.org>.
GitHub user erickguan added a comment to the discussion: discuss: Remove Metakey Concept
Thanks for working on this!
GitHub link: https://github.com/apache/opendal/discussions/5287#discussioncomment-11229752
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: discuss: Remove Metakey Concept
Posted by "erickguan (via GitHub)" <gi...@apache.org>.
GitHub user erickguan added a comment to the discussion: discuss: Remove Metakey Concept
Looks good to me. This would simplify service lister implementations.
I can see streaming API fetching metadata. I am interested in how you will solve asynchronous operations!
GitHub link: https://github.com/apache/opendal/discussions/5287#discussioncomment-11165398
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org