You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by xu...@apache.org on 2026/06/17 16:56:17 UTC
(opendal) 01/01: docs: clarify operator context composition
This is an automated email from the ASF dual-hosted git repository.
Xuanwo pushed a commit to branch xuanwo/operator-api-docs
in repository https://gitbox.apache.org/repos/asf/opendal.git
commit f88cbb11f3e03ca3d845d45db11f3489d25f7d20
Author: Xuanwo <gi...@xuanwo.io>
AuthorDate: Thu Jun 18 00:55:59 2026 +0800
docs: clarify operator context composition
---
core/core/src/docs/concepts.rs | 117 ++++++++++++++-------
core/core/src/docs/internals/layer.rs | 42 +++++---
.../core/src/docs/performance/http_optimization.md | 12 ++-
core/core/src/raw/layer.rs | 5 +
core/core/src/types/context/operation.rs | 16 ++-
core/core/src/types/operator/operator.rs | 77 +++++++++++---
6 files changed, 196 insertions(+), 73 deletions(-)
diff --git a/core/core/src/docs/concepts.rs b/core/core/src/docs/concepts.rs
index a18a7a504..29383df77 100644
--- a/core/core/src/docs/concepts.rs
+++ b/core/core/src/docs/concepts.rs
@@ -17,40 +17,30 @@
//! The core concepts of OpenDAL's public API.
//!
-//! OpenDAL provides a unified abstraction that helps developers access all storage services.
+//! OpenDAL provides a unified abstraction for accessing storage services.
//!
-//! There are two core concepts in OpenDAL:
+//! Public users mostly work with three concepts:
//!
-//! - [`Builder`]: Builder accepts a series of parameters to set up an instance of underlying services.
-//! You can adjust the behaviour of underlying services with these parameters.
-//! - [`Operator`]: Developer can access underlying storage services with manipulating one Operator.
-//! The Operator is a delegate for underlying implementation detail, and provides one unified access interface,
-//! including `read`, `write`, `list` and so on.
+//! - [`Builder`]: configures and builds a storage service.
+//! - [`Operator`]: owns a service stack and exposes storage operations such as
+//! `read`, `write`, and `list`.
+//! - [`Layer`][crate::raw::Layer]: wraps an operator's service stack or runtime
+//! resources to add behavior such as retry, timeout, tracing, and metrics.
//!
//! If you are interested in internal implementation details, please have a look at [`internals`][super::internals].
//!
//! # Builder
//!
-//! Let's start with [`Builder`].
-//!
-//! A `Builder` is a trait that is implemented by the underlying services. We can use a `Builder` to configure and create a service.
-//! Developer can only create one service via Builder, in other words, Builder is the only public API provided by services.
-//! And other detailed implementation will be hidden.
+//! A [`Builder`] configures and creates the underlying storage service. Service
+//! crates expose builders through [`services`][crate::services], for example
+//! [`services::Memory`][crate::services::Memory].
//!
//! ```text
-//! ┌───────────┐ ┌───────────┐
-//! │ │ build() │ │
-//! │ Builder ├────────────────►│ Service │
-//! │ │ │ │
-//! └───────────┘ └───────────┘
+//! ┌─────────┐ build() ┌─────────┐
+//! │ Builder ├────────────►│ Service │
+//! └─────────┘ └─────────┘
//! ```
//!
-//! All [`Builder`] provided by OpenDAL is under [`services`][crate::services], we can refer to them like `opendal::services::Memory`.
-//! By right the builder will be named like `OneServiceBuilder`, but usually we will export it to public with renaming it as one
-//! general name. For example, we will rename `S3Builder` to `S3` and developer will use `S3` finally.
-//!
-//! For example:
-//!
//! ```no_run
//! use opendal_core::services::Memory;
//!
@@ -58,22 +48,27 @@
//! ```
//!
//! # Operator
-//! The [`Operator`] is a delegate for Service, the underlying implementation detail that implements [`Service`][crate::raw::Service],
-//! and it also provides one unified access interface.
-//! It will hold one reference of Service with its all generic types erased by OpenDAL,
-//! which is the reason why we say the Operator is the delegate of one Service.
+//!
+//! An [`Operator`] is the public handle for a storage service stack. It stores:
+//!
+//! - a base service created from the builder.
+//! - a base [`OperationContext`] with runtime resources such as HTTP transport
+//! and executor.
+//! - an ordered list of layers.
+//! - the composed service and context used by storage operations.
//!
//! ```text
-//! ┌────────────────────┐
-//! │ Operator │
-//! │ │delegate │
-//! ┌─────────┐ build │ ▼ │ rely on ┌─────────────────────┐
-//! │ Builder ├───────┼──►┌────────────┐ │◄────────┤ business logic code │
-//! └─────────┘ │ │ Service │ │ └─────────────────────┘
-//! └───┴────────────┴───┘
+//! ┌─────────┐ Operator::new ┌────────────────────────────┐
+//! │ Builder ├──────────────────►│ Operator │
+//! └─────────┘ │ - base service │
+//! │ - base OperationContext │
+//! │ - layers │
+//! │ - composed service/context │
+//! └────────────────────────────┘
//! ```
//!
-//! `Operator` can be built from `Builder`:
+//! `Operator::new` returns a ready-to-use operator. There is no separate
+//! `finish` step.
//!
//! ```no_run
//! # use opendal_core::Result;
@@ -88,12 +83,55 @@
//! # }
//! ```
//!
-//! - `Operator` has it's internal `Arc`, so it's **cheap** to clone it.
-//! - `Operator` doesn't have generic parameters or lifetimes, so it's **easy** to use it everywhere.
-//! - `Operator` implements `Send` and `Sync`, so it's **safe** to send it between threads.
+//! - `Operator` is cheap to clone because it stores shared handles internally.
+//! - `Operator` has no generic parameters or lifetimes, so it is easy to pass
+//! through application code.
+//! - `Operator` is `Send` and `Sync`, so it can be shared across threads.
+//! - Methods that change layers or runtime resources return a new operator;
+//! existing clones and in-flight operations keep using their current composed
+//! service and context.
+//!
+//! # Runtime resources and layers
+//!
+//! [`OperationContext`] carries runtime resources from the operator to services,
+//! such as [`HttpTransporter`][crate::HttpTransporter] and [`Executor`][crate::Executor].
+//! Operation arguments such as ranges, versions, and concurrency limits stay in
+//! the `Op*` argument structs used by each operation.
+//!
+//! Use [`Operator::with_context`] to replace the base context. Use
+//! [`Operator::layer`] to append a layer. In both cases, OpenDAL replays the full
+//! layer list from the base service and base context to produce a fresh composed
+//! service and context.
+//!
+//! ```text
+//! base service ─┐
+//! ├─ layer replay ─► composed service ─┐
+//! base context ─┘ ├─► operation dispatch
+//! │ srv.read(&ctx, ...)
+//! ▼
+//! composed context
+//! ```
+//!
+//! ```no_run
+//! # use opendal_core::Result;
+//! use opendal_core::HttpTransporter;
+//! use opendal_core::OperationContext;
+//! use opendal_core::Operator;
+//! use opendal_core::services::Memory;
+//!
+//! # fn test() -> Result<()> {
+//! let transport = HttpTransporter::default();
+//! let op = Operator::new(Memory::default())?.with_context(
+//! OperationContext::new().with_http_transport(transport),
+//! );
+//! # let _ = op;
+//! # Ok(())
+//! # }
+//! ```
//!
-//! After get an `Operator`, we can do operations on different paths.
+//! # Operations
//!
+//! After creating an operator, use it to run operations on normalized paths.
//!
//! ```text
//! ┌──────────────┐
@@ -127,3 +165,4 @@
//!
//! [`Builder`]: crate::Builder
//! [`Operator`]: crate::Operator
+//! [`OperationContext`]: crate::OperationContext
diff --git a/core/core/src/docs/internals/layer.rs b/core/core/src/docs/internals/layer.rs
index 619d566d2..a45f6b07b 100644
--- a/core/core/src/docs/internals/layer.rs
+++ b/core/core/src/docs/internals/layer.rs
@@ -20,7 +20,7 @@
//! OpenDAL has one layer composition surface:
//!
//! - [`Layer`] receives an already erased [`ServiceDyn`] stack and can wrap the
-//! service, HTTP fetcher, or executor.
+//! service or its [`OperationContext`].
//! - Layer wrappers still implement typed [`Service`], so their own reader,
//! writer, lister, deleter, and copier bodies stay concrete.
//! - The composition boundary is [`Servicer`]. Boxing operation
@@ -28,22 +28,20 @@
//! for typed [`Service`] values.
//!
//! [`Layer`] itself is the runtime hook surface. Every hook returns `inner` by
-//! default, so a layer only implements the resource planes it needs:
+//! default, so a layer only implements the plane it needs:
//!
//! ```ignore
//! pub trait Layer: Send + Sync + Debug + Unpin + 'static {
//! fn apply_service(&self, srv: Servicer) -> Servicer;
-//! fn apply_http_transport(&self, srv: Servicer, inner: HttpTransporter) -> HttpTransporter;
-//! fn apply_execute(&self, srv: Servicer, inner: Executor) -> Executor;
+//! fn apply_context(&self, srv: Servicer, inner: OperationContext) -> OperationContext;
//! }
//! ```
//!
-//! [`Operator`] replays the same ordered layer list for each resource plane:
-//! first the operation service stack, then the HTTP fetch stack and task
-//! executor. Resource hooks receive the final [`Servicer`], so they can observe
-//! service identity without smuggling it through request extensions. The
-//! composed HTTP fetcher and executor are then stored in [`OperationContext`]
-//! and passed to service operations.
+//! [`Operator`] replays the same ordered layer list over the base service and
+//! base [`OperationContext`]. It first composes the operation service stack.
+//! Then context hooks receive the final [`Servicer`] and compose runtime
+//! resources such as HTTP transport and executor. The composed context is passed
+//! to service operations.
//!
//! An operation layer normally has two parts:
//!
@@ -66,11 +64,25 @@
//! forwarding calls to [`ServiceDyn`], while the wrapper keeps concrete
//! operation body types until it is returned as a [`Servicer`].
//!
-//! Resource-only layers can implement only `apply_http_transport` or
-//! `apply_execute`. If they do not need the final service stack, they should
-//! name that parameter `_srv`. Layers that need consistent policy across planes
-//! can implement multiple hooks, for example operation and HTTP concurrency
-//! limits or operation and I/O timeout handling.
+//! Resource-only layers can implement only `apply_context`. If they do not need
+//! the final service stack, they should name that parameter `_srv`.
+//!
+//! ```ignore
+//! pub struct TransportLayer {
+//! transport: HttpTransporter,
+//! }
+//!
+//! impl Layer for TransportLayer {
+//! fn apply_context(&self, _srv: Servicer, inner: OperationContext) -> OperationContext {
+//! inner.with_http_transport(self.transport.clone())
+//! }
+//! }
+//! ```
+//!
+//! Layers that need consistent policy across planes can implement both hooks,
+//! for example operation and I/O timeout handling. Resource wrappers that
+//! replace HTTP transport or executor must forward to the previous value when
+//! they want lower layers to remain effective.
//!
//! [`Layer`]: crate::raw::Layer
//! [`Service`]: crate::raw::Service
diff --git a/core/core/src/docs/performance/http_optimization.md b/core/core/src/docs/performance/http_optimization.md
index 9a420e496..029f410d3 100644
--- a/core/core/src/docs/performance/http_optimization.md
+++ b/core/core/src/docs/performance/http_optimization.md
@@ -26,7 +26,8 @@ let client = reqwest::ClientBuilder::new()
.expect("http client must be created");
// Replace the operator's base HTTP transport.
-let op = op.http_transport(HttpTransporter::new(ReqwestTransport::new(client)));
+let transport = HttpTransporter::new(ReqwestTransport::new(client));
+let op = op.with_context(OperationContext::new().with_http_transport(transport));
```
## DNS Caching
@@ -45,7 +46,8 @@ let client = reqwest::ClientBuilder::new()
.expect("http client must be created");
// Replace the operator's base HTTP transport.
-let op = op.http_transport(HttpTransporter::new(ReqwestTransport::new(client)));
+let transport = HttpTransporter::new(ReqwestTransport::new(client));
+let op = op.with_context(OperationContext::new().with_http_transport(transport));
```
The default DNS cache settings from `hickory_dns` are generally sufficient for most workloads. However, if you have specific requirements—such as sharing the same DNS cache across multiple HTTP clients or configuring the DNS cache size—you can use the `Xuanwo/reqwest-hickory-resolver` crate to set up a custom DNS resolver.
@@ -78,7 +80,8 @@ let client = reqwest::ClientBuilder::new()
.expect("http client must be created");
// Replace the operator's base HTTP transport.
-let op = op.http_transport(HttpTransporter::new(ReqwestTransport::new(client)));
+let transport = HttpTransporter::new(ReqwestTransport::new(client));
+let op = op.with_context(OperationContext::new().with_http_transport(transport));
```
The `ResolverOpts` has many options that can be configured. For a complete list of options, please refer to the [hickory_resolver documentation](https://docs.rs/hickory-resolver/latest/hickory_resolver/config/struct.ResolverOpts.html).
@@ -105,7 +108,8 @@ let client = reqwest::ClientBuilder::new()
.expect("http client must be created");
// Replace the operator's base HTTP transport.
-let op = op.http_transport(HttpTransporter::new(ReqwestTransport::new(client)));
+let transport = HttpTransporter::new(ReqwestTransport::new(client));
+let op = op.with_context(OperationContext::new().with_http_transport(transport));
```
It's also recommended to use opendal's [`TimeoutLayer`][crate::layers::TimeoutLayer] to prevent slow requests hangs forever. This layer will automatically cancel the request if it takes too long to complete.
diff --git a/core/core/src/raw/layer.rs b/core/core/src/raw/layer.rs
index 646936a07..ee2226933 100644
--- a/core/core/src/raw/layer.rs
+++ b/core/core/src/raw/layer.rs
@@ -27,6 +27,11 @@ use crate::*;
/// the operation service, the operation context (HTTP transport and executor),
/// or both.
///
+/// [`Operator`][crate::Operator] applies layers by first composing the service
+/// stack with [`Layer::apply_service`], then composing [`OperationContext`] with
+/// [`Layer::apply_context`]. The context hook receives the final service stack
+/// so resource wrappers can observe service identity or capability when needed.
+///
/// Hooks take `&self`, so layers that keep mutable state must use interior
/// mutability. That state must remain `Send` and `Sync` because layers are
/// shared across cloned operators and concurrent operations.
diff --git a/core/core/src/types/context/operation.rs b/core/core/src/types/context/operation.rs
index 48f138406..cee093c56 100644
--- a/core/core/src/types/context/operation.rs
+++ b/core/core/src/types/context/operation.rs
@@ -19,6 +19,18 @@ use crate::*;
/// Composed resources passed from operator to services and layers.
///
+/// [`Operator`][crate::Operator] keeps a base `OperationContext` and replays its
+/// layers to produce the composed context passed to every service operation.
+/// The context carries runtime resources supplied by the operator stack, not
+/// caller intent for a specific operation. Operation-specific inputs live in
+/// `Op*` argument structs such as `OpRead` and `OpWrite`.
+///
+/// Service implementations should read HTTP transport, executor, and similar
+/// resources from the `OperationContext` they receive at the operation boundary.
+/// They should not cache those resources in the service created by
+/// [`Builder::build`], because later layers or [`Operator::with_context`] can
+/// replace them for a derived operator.
+///
/// Layers that replace the HTTP transport or executor must keep forwarding
/// requests or tasks to the previous value. Otherwise, composed features such as
/// retry, timeout, tracing, and metrics can be bypassed.
@@ -29,12 +41,12 @@ pub struct OperationContext {
}
impl OperationContext {
- /// Create a new operation context with default resources.
+ /// Create a new operation context with default runtime resources.
pub fn new() -> Self {
Self::from_parts(HttpTransporter::default(), Executor::default())
}
- /// Create a new operation context from composed resources.
+ /// Create a new operation context from composed runtime resources.
pub fn from_parts(http_transport: HttpTransporter, executor: Executor) -> Self {
Self {
http_transport,
diff --git a/core/core/src/types/operator/operator.rs b/core/core/src/types/operator/operator.rs
index 693f0efc7..02e5eb7ad 100644
--- a/core/core/src/types/operator/operator.rs
+++ b/core/core/src/types/operator/operator.rs
@@ -29,12 +29,27 @@ use crate::*;
/// The `Operator` serves as the entry point for all public asynchronous APIs.
///
-/// For more details about the `Operator`, refer to the [`concepts`][crate::docs::concepts] section.
+/// For more details about the `Operator`, refer to the concepts section in the
+/// crate documentation.
///
/// `Operator` is immutable: methods that change layers, HTTP transport, or
/// executor return a new operator. Existing clones and in-flight operations keep
/// using the service stack and operation context they already hold.
///
+/// Internally, an operator keeps base providers and composed dispatch state:
+///
+/// | Method | Meaning |
+/// | --- | --- |
+/// | [`Operator::base_service`] | The storage service before user layers are applied. |
+/// | [`Operator::base_context`] | The runtime resources before user layers are applied. |
+/// | [`Operator::service`] | The storage service stack that receives `read`, `write`, `list`, and other operations. |
+/// | [`Operator::context`] | The runtime resources passed with those operations. |
+///
+/// [`Operator::layer`] appends a layer and replays all layers from the base
+/// service and base context. [`Operator::with_context`] replaces the base
+/// context and then performs the same replay. This keeps the composed service
+/// and context from drifting apart.
+///
/// ## Build
///
/// Users can initialize an `Operator` through the following methods:
@@ -55,15 +70,21 @@ use crate::*;
/// }
/// ```
///
-/// ## Layer
+/// ## Runtime Resources And Layers
///
-/// After the operator is built, users can add the layers they need on top of it.
+/// After the operator is built, users can replace runtime resources or add
+/// layers on top of it.
///
/// OpenDAL offers various layers for users to choose from. Visit [`layers`] for further details.
///
-/// Layers are replayed from the operator's base service, HTTP transport, and
-/// executor whenever the operator is changed. Cloned operators can therefore
-/// add layers or replace providers independently.
+/// Runtime resources are stored in [`OperationContext`]. HTTP based services
+/// receive the composed context for each operation and use its HTTP transport
+/// and executor instead of caching those resources in the service built by the
+/// builder.
+///
+/// Layers are replayed from the operator's base service and base context
+/// whenever the operator is changed. Cloned operators can therefore add layers
+/// or replace providers independently.
///
/// ```
/// use opendal_core::HttpTransporter;
@@ -175,10 +196,14 @@ impl Operator {
Self::from_parts(OperationContext::default(), srv)
}
- /// Build an operator from its composed `ctx` and `service`.
+ /// Build an operator from a pre-composed `ctx` and `service`.
+ ///
+ /// This is the low-level constructor for callers that already have a
+ /// [`Servicer`] and [`OperationContext`]. Most users should use
+ /// [`Operator::new`] instead so OpenDAL can install its default layers.
///
- /// Pairs with [`Operator::into_parts`]. The operator starts with no layers;
- /// `ctx` and `service` are used directly for dispatch.
+ /// Pairs with [`Operator::into_parts`]. The operator starts with no layers,
+ /// so `ctx` and `service` are also the composed dispatch state.
pub fn from_parts(ctx: OperationContext, srv: Servicer) -> Self {
Self {
base_srv: srv.clone(),
@@ -190,6 +215,10 @@ impl Operator {
}
/// Split the operator into its composed `ctx` and `service` for direct dispatch.
+ ///
+ /// Base providers and layer replay history are discarded. Reconstructing an
+ /// operator with [`Operator::from_parts`] uses the returned values directly
+ /// and starts with an empty layer list.
pub fn into_parts(self) -> (OperationContext, Servicer) {
(self.ctx, self.srv)
}
@@ -213,22 +242,36 @@ impl Operator {
(srv, ctx)
}
- /// Get the base `srv` that layers are applied to.
+ /// Get the storage service before user layers are applied.
+ ///
+ /// This is useful for code that needs to inspect the original backend. Most
+ /// operation code should use [`Operator::service`] instead, because that is
+ /// the stack that includes layers.
pub fn base_service(&self) -> &Servicer {
&self.base_srv
}
- /// Get the base `ctx` that layers are applied to.
+ /// Get the operation context before user layers are applied.
+ ///
+ /// This is useful for code that needs to inspect the original runtime
+ /// resources. Most operation code should use [`Operator::context`] instead,
+ /// because that includes context changes made by layers.
pub fn base_context(&self) -> &OperationContext {
&self.base_ctx
}
- /// Get the composed `srv` used by the current operator.
+ /// Get the storage service stack that receives operations.
+ ///
+ /// This stack includes the base service plus all user layers applied to the
+ /// operator.
pub fn service(&self) -> &Servicer {
&self.srv
}
- /// Get the composed `ctx` passed to `srv` for operations.
+ /// Get the runtime resources passed with operations.
+ ///
+ /// This context includes the base context plus all context changes made by
+ /// user layers.
pub fn context(&self) -> &OperationContext {
&self.ctx
}
@@ -254,6 +297,10 @@ impl Operator {
}
/// Replace the base operation context and rebuild composed state.
+ ///
+ /// Existing layers are preserved and replayed from the same base service
+ /// with the new base context. This is the public API for replacing runtime
+ /// resources such as HTTP transport or executor on an operator.
#[must_use]
pub fn with_context(self, ctx: OperationContext) -> Self {
let (srv, composed) = Self::apply_layers(&self.base_srv, &ctx, &self.layers);
@@ -267,6 +314,10 @@ impl Operator {
}
/// Apply a layer to this operator and rebuild composed state.
+ ///
+ /// The new layer is appended after existing layers. OpenDAL then replays all
+ /// layers from the base service and base context, so the composed service and
+ /// composed context are produced by the same ordered layer list.
#[must_use]
pub fn layer<L: Layer>(self, layer: L) -> Self {
let mut layers = Arc::unwrap_or_clone(self.layers);