[I] new feature: suggestions for `Lister` [opendal]

["chitralverma (via GitHub)" <gi...@apache.org>]

chitralverma opened a new issue, #6272:
URL: https://github.com/apache/opendal/issues/6272

   ### Feature Description
   
   Thought of a few `Lister` and `Entry` enhancements for your consideration.
   (Affects public API)
   
   
   
   
   ### Problem and Solution
   
   - `stat` returns `Metadata` which is okay, and `Lister` returns an `Entry` which is path +`Metadata`. But since `Lister` [doesn't guarantee metadata retrieval due to performance implications](https://github.com/apache/opendal/pull/6232#discussion_r2126959344), and Entry should be path +`Option<Metadata>` instead.
   - Some user might still be ready for those performance cost, and for this `ListOptions` should have a key `with_metadata: bool` which ensures the metadata retrieval. This way the user decides performance vs results. 
   (they can still do this with an external `stat` call but user side code simplifies with this simple new option).
   - Additionally, `stat` already has some options which might be useful in `Lister` as well (can be pushed down, if backend services support it),
     - if_match
     - if_none_match
     - if_modified_since
     - if_unmodified_since
   - Suggestions for better naming,
     - `deleted` -> `with_deleted`
     - `versions` -> `with_versions`
   
   what do you think?
   
   ### Additional Context
   
   _No response_
   
   ### Are you willing to contribute to the development of this feature?
   
   - [x] Yes, I am willing to contribute to the development of this feature.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] new feature: suggestions for `Lister` [opendal]

["Xuanwo (via GitHub)" <gi...@apache.org>]

Xuanwo commented on issue #6272:
URL: https://github.com/apache/opendal/issues/6272#issuecomment-2955386361

   Thank you @chitralverma for raising this.
   
    
   > * But since `Lister` [doesn't guarantee metadata retrieval due to performance implications](https://github.com/apache/opendal/pull/6232#discussion_r2126959344), and Entry should be path +`Option<Metadata>` instead.
   
   `Metadata` is always available, as we will provide at least the file type. Changing it to `Option<Metadata>` doesn't make sense to me.
   
   The more complex issue here is that different services may return different sets of metadata. For example, S3 might have an `etag` but not a `content_type`. So even if we return it as `Option<Metadata>`, users will still have to handle each metadata field on a case-by-case basis.
   
   > Some user might still be ready for those performance cost, and for this ListOptions should have a key with_metadata: bool which ensures the metadata retrieval. This way the user decides performance vs results.
   > (they can still do this with an external stat call but user side code simplifies with this simple new option).
   
   We had this before. We designed a complex system specifically for this purpose, but in reality, it doesn't simplify things. Having `list` overlap with `stat` is not a good idea. We discussed this in [opendal::docs::rfcs::rfc_5314_remove_metakey](https://docs.rs/opendal/latest/opendal/docs/rfcs/rfc_5314_remove_metakey/index.html).
   
   > (can be pushed down, if backend services support it),
   
   No services can push this down, except perhaps for certain in-memory structures. It might be easier for us to use a `filter_map` externally.
   
   > Suggestions for better naming,
   > `deleted` -> with_deleted
   > `versions` -> with_versions
   
   It seems good idea to me to change them into `includes_deleted` and `includes_versions`. How about create a seperate issue for this? 
   
   I picked up `deleted` and `versions` before since our API usage will look like:
   
   ```rust
   let res = op.list_with(path).deleted(true).versions(true).await?;
   ```
   
   Will this flavor cause confusion about our behavior?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org