Re: [PR] RFC-5444: Operator From Uri [opendal]

["jorgehermo9 (via GitHub)" <gi...@apache.org>]

jorgehermo9 commented on code in PR #5444:
URL: https://github.com/apache/opendal/pull/5444#discussion_r1896683002


##########
core/src/docs/rfcs/5444_operator_from_uri.md:
##########
@@ -0,0 +1,144 @@
+- Proposal Name: `operator_from_uri`
+- Start Date: 2024-12-23
+- RFC PR: [apache/opendal#5444](https://github.com/apache/opendal/pull/5444)
+- Tracking Issue: [apache/opendal#5445](https://github.com/apache/opendal/issues/5445)
+
+# Summary
+
+This RFC proposes adding URI-based configuration support to OpenDAL, allowing users to create operators directly from URIs. The proposal introduces a new `from_uri` API in both the `Operator` and `Configurator` traits, along with an `OperatorRegistry` to manage operator factories.
+
+# Motivation
+
+Currently, creating an operator in OpenDAL requires explicit configuration through builder patterns. While this approach provides type safety and clear documentation, it can be verbose and inflexible for simple use cases. Many storage systems are naturally identified by URIs (e.g., `s3://bucket/path`, `fs:///path/to/dir`).
+
+Adding URI-based configuration would:
+
+- Simplify operator creation for common use cases
+- Enable configuration via connection strings (common in many applications)
+- Make OpenDAL more approachable for new users
+- Allow dynamic operator creation based on runtime configuration
+
+# Guide-level explanation
+
+The new API allows creating operators directly from URIs:
+
+```rust
+// Create an operator using URI
+let op = Operator::from_uri("s3://my-bucket/path", vec![
+    ("access_key_id".to_string(), "xxx".to_string()),
+    ("secret_key_key".to_string(), "yyy".to_string()),
+])?;
+
+// Create a file system operator
+let op = Operator::from_uri("fs:///tmp/test", vec![])?;
+
+// Using with custom registry
+let registry = OperatorRegistry::new();
+registry.register("custom", my_factory);
+let op = registry.parse("custom://endpoint", options)?;
+```
+
+# Reference-level explanation
+
+The implementation consists of three main components:
+
+1. The `OperatorRegistry`:
+
+```rust
+type OperatorFactory = fn(http::Uri, HashMap<String, String>) -> Result<Operator>;
+
+pub struct OperatorRegistry {
+    register: Arc<Mutex<HashMap<String, OperatorFactory>>>,
+}
+
+impl OperatorRegistry {
+    fn register(&self, scheme: &str, factory: OperatorFactory) {
+        ...
+    }
+
+    fn parse(&self, uri: &str, options: impl IntoIterator<Item = (String, String)>) -> Result<Operator> {
+        ...
+    }
+}
+```
+
+2. The `Configurator` trait extension:
+
+```rust
+impl Configurator for S3Config {
+    fn from_uri(uri: &str, options: impl IntoIterator<Item = (String, String)>) -> Result<Self> {
+        ...
+    }
+}
+```
+
+3. The `Operator` factory method:
+
+```rust
+impl Operator {
+    pub fn from_uri(
+        uri: &str,
+        options: impl IntoIterator<Item = (String, String)>,
+    ) -> Result<Self> {
+        static REGISTRY: Lazy<OperatorRegistry> = Lazy::new(|| {
+            let registry = OperatorRegistry::new();
+            // Register built-in operators
+            registry.register("s3", s3_factory);
+            registry.register("fs", fs_factory);
+            // ...
+            registry
+        });
+      
+        REGISTRY.parse(uri, options)
+    }
+}
+```
+
+We are intentionally using `&str` instead of `Scheme` here to simplify working with external components outside this crate. Additionally, we plan to remove `Scheme` from our public API soon to enable splitting OpenDAL into multiple crates.
+
+# Drawbacks
+
+- Increases API surface area
+- Less type safety compared to builder patterns
+- Potential for confusing error messages with invalid URIs
+- Need to maintain backwards compatibility
+
+# Rationale and alternatives
+
+Alternatives considered:
+
+1. Connection string format instead of URIs
+2. Builder pattern with URI parsing
+3. Macro-based configuration
+
+URI-based configuration was chosen because:

Review Comment:
   I think that a major reason should be that users can switch storage backends with no code change. Operator backends changes would be *a lot easier* and users can try a lot of backends with nearly no effort



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org