[I] new feature: Allow layers to be placed before `SimulateLayer` / `CompleteLayer` [opendal]

["ngg (via GitHub)" <gi...@apache.org>]

ngg opened a new issue, #7783:
URL: https://github.com/apache/opendal/issues/7783

   ### Feature Description
   
   Add an API that lets advanced users build an operator layer stack where a custom
   layer can be placed between the base service and OpenDAL's internal default
   layers, especially `SimulateLayer` and `CompleteLayer`.
   
   I'm writing a complex `Layer` that can turn a single user operation into
   multiple lower-level service calls. To do this efficiently, the layer needs to
   know the native capabilities of the underlying base service, not capabilities
   added later by simulation/completion layers.
   
   What I'd like is the ability to place my layer between the base service and
   `SimulateLayer` / `CompleteLayer`.
   
   ### Problem and Solution
   
   With the [new `Capability` API](https://github.com/apache/opendal/pull/7747),
   this is difficult because `Operator::new` / URI construction automatically
   applies internal layers such as `ErrorContextLayer`, `SimulateLayer`,
   `CompleteLayer`, and `CorrectnessCheckLayer`. By the time my layer sees the
   inner service, capabilities may already include operations provided by
   `SimulateLayer` / `CompleteLayer`, so the layer can't reliably distinguish
   native support from simulated support.
   
   The Capability RFC mentions a `pub(crate)` or raw `service_capability()` function. If that
   existed and were public, it would also help with this problem. However, the
   actual implementation doesn't seem to have added that function; instead, there
   is a single `Capability` list. I agree that a single `Capability` list is
   cleaner, but then complex layers need another way to reason about what the base
   service supports natively.
   
   Possible solutions:
   
   - Add an `OperatorRegistry` API to load only the base service from a URI, for
     example:
   
     ```rust
     pub fn load_service(&self, uri: impl IntoOperatorUri) -> Result<Servicer>
     ```
   
   - Or add an `Operator` API to expose the base service and base context from a
     built operator, so callers can rebuild it with a custom layer order.
   
     ```rust
     pub fn base_service(&self) -> &Servicer
     pub fn base_context(&self) -> &OperationContext
     ```
   
   With either option, callers could use `Operator::from_parts` to build an
   operator from the base service, add their own layers, and then add the internal
   default layers at the end.
   
   This would let complex layers optimize against true native backend capabilities
   while still allowing callers to add simulation/completion/checking layers
   outside them.
   
   One caveat: this only fully works for layers that don't need to modify
   `OperationContext`. If the only way to add the internal default layers is still
   `Operator::new`, then the operator is rebuilt with `OperationContext::default()`,
   which loses any context changes from earlier layers.
   
   Possible ways to solve that:
   
   - Make the internal default layers public. This is the most flexible option:
     users could apply them manually in any order, and could also place something
     like `CorrectnessCheckLayer` below their complex layer to catch bugs in that
     layer.
   - Or add an `Operator` method that applies the internal default layers to an
     existing operator while preserving its base context and composed layer stack,
     for example:
   
     ```rust
     pub fn add_internal_layers(self) -> Self
     ```
   
     Then callers could use `Operator::from_parts`, add their own layers, and call
     `add_internal_layers()` at the end.
   - Or change the `Builder` trait so `build` returns both the service and the
     `OperationContext` that should be used with it.
   
     ```rust
     fn build(self) -> Result<(impl Service, OperationContext)>
     ```
   
   ### Additional Context
   
   _No response_
   
   ### Are you willing to contribute to the development of this feature?
   
   - [x] Yes, I am willing to contribute to the development of this feature.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] new feature: Allow layers to be placed before `SimulateLayer` / `CompleteLayer` [opendal]

["ngg (via GitHub)" <gi...@apache.org>]

ngg commented on issue #7783:
URL: https://github.com/apache/opendal/issues/7783#issuecomment-4732707316

   I've opened a pull request with the smaller change. Let me know if you'd prefer the other option to modify OperatorRegistry instead, or if you have a better idea.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] new feature: Allow layers to be placed before `SimulateLayer` / `CompleteLayer` [opendal]

["Xuanwo (via GitHub)" <gi...@apache.org>]

Xuanwo closed issue #7783: new feature: Allow layers to be placed before `SimulateLayer` / `CompleteLayer`
URL: https://github.com/apache/opendal/issues/7783


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org