You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "XuQianJin-Stars (via GitHub)" <gi...@apache.org> on 2026/04/27 03:19:53 UTC
[PR] feat(services/goosefs): implement opendal-service-goosefs crate backed by goosefs-sdk [opendal]
XuQianJin-Stars opened a new pull request, #7428:
URL: https://github.com/apache/opendal/pull/7428
# Which issue does this PR close?
Closes #7427
# Rationale for this change
GooseFS is a distributed caching file system developed by Tencent Cloud. Although it is API-compatible with Alluxio at the REST-proxy level, using the proxy adds an extra network hop and prevents block-level streaming I/O.
This PR introduces a **native gRPC** GooseFS backend for OpenDAL that talks directly to the GooseFS **Master (`:9200`)** for metadata and to **Workers (`:9203`)** for block I/O, eliminating the proxy component and enabling efficient large-file transfer through bidirectional streaming.
It is implemented as a new standalone crate `opendal-service-goosefs`, wrapping the official [`goosefs-sdk`](https://crates.io/crates/goosefs-sdk) `0.1.0` gRPC client, and exposes the standard OpenDAL `Access` trait so that users can access GooseFS clusters through the unified OpenDAL API.
# What changes are included in this PR?
## New crate: `core/services/goosefs/`
11 new files implementing the full service:
| File | Responsibility |
|---|---|
| `src/lib.rs` | Crate entry & re-exports |
| `src/backend.rs` | `GooseFsBuilder` + `GooseFsBackend` (`Access` impl) |
| `src/config.rs` | `GooseFsConfig` (connection / auth / write-type) |
| `src/core.rs` | gRPC client core, master/worker routing |
| `src/error.rs` | gRPC `Status` → OpenDAL `Error` mapping |
| `src/reader.rs` | Block-level streaming reader |
| `src/writer.rs` | Block-level streaming writer |
| `src/lister.rs` | Directory listing |
| `src/deleter.rs` | File / directory deletion |
| `src/docs.md` | Service documentation |
| `Cargo.toml` | Crate manifest, depends on `goosefs-sdk = "0.1.0"` |
## Core registration
- `core/Cargo.toml` — add `services-goosefs` feature and optional dependency on `opendal-service-goosefs`.
- `core/src/services/mod.rs` — register the new `goosefs` service behind the feature flag.
## Key design points
- **Native gRPC protocol** — direct connection to Master `:9200` (metadata) and Worker `:9203` (block I/O); no Proxy required.
- **HA multi-master support** — `master_addr` accepts a comma-separated list; primary Master is discovered automatically.
- **Consistent-hash worker routing** — worker selection hashes on block IDs for data locality.
- **Four write types** — `MUST_CACHE`, `CACHE_THROUGH`, `THROUGH`, `ASYNC_THROUGH`.
- **Block-level streaming read/write** — implemented via gRPC **bidirectional streams** for efficient large-file transfer.
- **Authentication** — `SIMPLE` and `NOSASL` modes; username is configurable (defaults to current OS user).
- **Cross-process-safe `FileSystemContext`** — PID-aware, automatically reconnects after `fork()` / `spawn()`.
- **Authentication-failure recovery** — `reset_ctx()` + one-shot reconnect on auth errors.
## Capabilities
- [x] `create_dir`
- [x] `stat`
- [x] `read`
- [x] `write`
- [x] `delete`
- [x] `list`
- [x] `rename`
# Are there any user-facing changes?
Yes — a new public service `services::GooseFs` is added, gated behind the `services-goosefs` feature flag. **No breaking changes** to existing APIs.
## Configuration
| Field | Description | Default |
|---|---|---|
| `root` | Working directory for the backend | `/` |
| `master_addr` | GooseFS Master `host:port` (comma-separated for HA) | — *(required)* |
| `block_size` | Block size for new files | `64 MiB` |
| `chunk_size` | Chunk size for streaming RPCs | `1 MiB` |
| `write_type` | `must_cache` / `cache_through` / `through` / `async_through` | `cache_through` |
| `auth_type` | `nosasl` / `simple` | `simple` |
| `auth_username` | Username for SIMPLE auth | current OS user |
## Usage
```
rust use opendal::Operator; use opendal::services::GooseFs;
let builder = GooseFs::default() .root("/data") .master_addr("10.0.0.1:9200,10.0.0.2:9200,10.0.0.3:9200") .write_type("cache_through");
let op = Operator::new(builder)?.finish();
```
Or via URI:
```
rust let op = Operator::from_uri("goosefs://10.0.0.1:9200/data")?;
```
# AI Usage Statement
This PR was developed with the assistance of AI coding tools (Claude, used for code scaffolding, documentation drafting, and review). All generated code has been manually reviewed, compiled, tested (`cargo build` / `cargo test` / `cargo clippy`), and verified against a live GooseFS cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/goosefs): implement opendal-service-goosefs crate backed by goosefs-sdk [opendal]
Posted by "XuQianJin-Stars (via GitHub)" <gi...@apache.org>.
XuQianJin-Stars commented on code in PR #7428:
URL: https://github.com/apache/opendal/pull/7428#discussion_r3201724061
##########
core/services/goosefs/src/backend.rs:
##########
@@ -0,0 +1,388 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+use std::sync::Arc;
+
+use log::debug;
+
+use super::GOOSEFS_SCHEME;
+use super::config::GooseFsConfig;
+use super::core::GooseFsCore;
+use super::deleter::GooseFsDeleter;
+use super::lister::GooseFsLister;
+use super::reader::GooseFsReader;
+use super::writer::GooseFsWriter;
+use super::writer::GooseFsWriters;
+use opendal_core::raw::*;
+use opendal_core::*;
+
+/// [GooseFS](https://cloud.tencent.com/product/goosefs) services support via native gRPC.
+#[doc = include_str!("docs.md")]
+#[derive(Default)]
+pub struct GooseFsBuilder {
+ pub(super) config: GooseFsConfig,
+}
+
+impl Debug for GooseFsBuilder {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("GooseFsBuilder")
+ .field("config", &self.config)
+ .finish_non_exhaustive()
+ }
+}
+
+impl GooseFsBuilder {
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(mut self, root: &str) -> Self {
+ self.config.root = if root.is_empty() {
+ None
+ } else {
+ Some(root.to_string())
+ };
+ self
+ }
+
+ /// Set master address(es).
+ ///
+ /// Single master: `"10.0.0.1:9200"`
+ /// HA (comma-separated): `"10.0.0.1:9200,10.0.0.2:9200,10.0.0.3:9200"`
+ ///
+ /// When provided here it **overrides** any address discovered from
+ /// `goosefs-site.properties` or `GOOSEFS_MASTER_ADDR`. When left unset
+ /// the builder falls back to the SDK's auto-discovery chain
+ /// (defaults → properties → env) — see `goosefs-sdk`
+ /// `docs/CLIENT_CONFIGURATION.md` §1. `build()` fails with
+ /// `ConfigInvalid` only if **no** source supplies a master address.
+ pub fn master_addr(mut self, addr: &str) -> Self {
+ if !addr.is_empty() {
+ self.config.master_addr = Some(addr.to_string());
+ }
+ self
+ }
+
+ /// Set block size for new files (bytes).
+ pub fn block_size(mut self, size: u64) -> Self {
+ self.config.block_size = Some(size);
+ self
+ }
+
+ /// Set chunk size for streaming RPCs (bytes).
+ pub fn chunk_size(mut self, size: u64) -> Self {
+ self.config.chunk_size = Some(size);
+ self
+ }
+
+ /// Set default write type.
+ ///
+ /// Values: `"must_cache"`, `"cache_through"`, `"through"`, `"async_through"`
+ pub fn write_type(mut self, wt: &str) -> Self {
+ if !wt.is_empty() {
+ self.config.write_type = Some(wt.to_string());
+ }
+ self
+ }
+
+ /// Set authentication type.
+ ///
+ /// Values: `"nosasl"`, `"simple"` (default: `"simple"`).
+ /// - `"nosasl"` — skip SASL authentication entirely.
+ /// - `"simple"` — PLAIN SASL with username (server does not verify password).
+ pub fn auth_type(mut self, auth_type: &str) -> Self {
+ if !auth_type.is_empty() {
+ self.config.auth_type = Some(auth_type.to_string());
+ }
+ self
+ }
+
+ /// Set authentication username.
+ ///
+ /// Used in SIMPLE mode as the login identity.
+ /// Default: current OS user (`$USER` / `$USERNAME`).
+ pub fn auth_username(mut self, username: &str) -> Self {
+ if !username.is_empty() {
+ self.config.auth_username = Some(username.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for GooseFsBuilder {
+ type Config = GooseFsConfig;
+
+ /// Build the backend and return a GooseFsBackend.
+ fn build(self) -> Result<impl Access> {
+ debug!("GooseFsBuilder::build started: {:?}", &self);
+
+ let root = normalize_root(&self.config.root.clone().unwrap_or_default());
+ debug!("GooseFsBuilder use root {}", &root);
+
+ // ── Step 1: establish the base SDK config ─────────────────────────────
+ //
+ // We follow the same priority chain that `FileSystemContext::connect`
+ // (and its `ConfigRefresher`) uses — see
+ // `docs/CLIENT_CONFIGURATION.md` §1 "Configuration Loading Priority":
+ //
+ // defaults < goosefs-site.properties < GOOSEFS_* env vars
+ //
+ // `GooseFsConfig::from_properties_auto()` already implements this
+ // chain and is the *same* function the SDK calls every 60s to refresh
+ // the transparent-acceleration switches. Using it here keeps the
+ // initial OpenDAL build and the in-process hot-reload semantically
+ // aligned — users who deploy `goosefs-site.properties` get the exact
+ // same config from both paths.
+ //
+ // Failure policy:
+ // * no properties file found → silently uses defaults + env
+ // (`from_properties_auto` handles this internally)
+ // * properties file present but malformed → hard-fail
+ // (broken config must not be silently dropped)
+ //
+ // Builder-explicit fields (Step 2) have the final say, overriding
+ // anything discovered from properties / env.
+ let mut goosefs_config = goosefs_sdk::config::GooseFsConfig::from_properties_auto()
+ .map_err(|e| {
+ Error::new(
+ ErrorKind::ConfigInvalid,
+ format!("failed to auto-load goosefs config: {e}"),
+ )
+ .with_operation("Builder::build")
+ .with_context("service", GOOSEFS_SCHEME)
+ })?;
+
+ // Root always comes from OpenDAL (it's an OpenDAL-layer concept).
+ goosefs_config.root = root.clone();
+
+ // ── Step 2: overlay builder-explicit fields (authoritative) ───────────
+ if let Some(ref master_addr) = self.config.master_addr {
+ let addrs: Vec<String> = master_addr
+ .split(',')
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .collect();
+
+ if addrs.is_empty() {
+ return Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ "master_addr is empty after trimming",
+ )
+ .with_operation("Builder::build")
+ .with_context("service", GOOSEFS_SCHEME));
+ }
+
+ if addrs.len() == 1 {
+ goosefs_config.master_addr = addrs[0].clone();
+ goosefs_config.master_addrs = Vec::new();
+ } else {
+ goosefs_config.master_addr = addrs[0].clone();
+ goosefs_config.master_addrs = addrs;
+ }
+ }
+
+ // After properties/env auto-load + builder overlay, we must still
+ // have at least one usable master address. If we don't, fail fast
+ // with a message that points the user at all three sources.
+ if goosefs_config.master_addr.is_empty() && goosefs_config.master_addrs.is_empty() {
+ return Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ "master_addr is not configured: set it via GooseFsBuilder::master_addr(...), \
+ the `master_addr` config key, the GOOSEFS_MASTER_ADDR env var, \
+ or `goosefs.master.hostname`/`goosefs.master.rpc.addresses` in goosefs-site.properties",
+ )
+ .with_operation("Builder::build")
+ .with_context("service", GOOSEFS_SCHEME));
+ }
+ debug!(
+ "GooseFsBuilder use master_addr {} (addrs={:?})",
+ &goosefs_config.master_addr, &goosefs_config.master_addrs
+ );
+
+ if let Some(block_size) = self.config.block_size {
+ goosefs_config.block_size = block_size;
+ }
+ if let Some(chunk_size) = self.config.chunk_size {
+ goosefs_config.chunk_size = chunk_size;
+ }
+
+ // Parse write_type string → goosefs_sdk::WritePType i32
+ if let Some(ref wt) = self.config.write_type {
+ let wt_i32 = match wt.as_str() {
+ "must_cache" | "MUST_CACHE" => 1,
+ "try_cache" | "TRY_CACHE" => 2,
+ "cache_through" | "CACHE_THROUGH" => 3,
+ "through" | "THROUGH" => 4,
+ "async_through" | "ASYNC_THROUGH" => 5,
+ _ => 1, // default to MUST_CACHE
+ };
+ goosefs_config.write_type = Some(wt_i32);
+ }
+
+ // Parse auth_type string → goosefs_sdk::auth::AuthType
+ if let Some(ref auth_type_str) = self.config.auth_type {
+ goosefs_config = goosefs_config
+ .with_auth_type_str(auth_type_str)
+ .map_err(|e| {
+ Error::new(
+ ErrorKind::ConfigInvalid,
+ format!("invalid auth_type: {}", e),
+ )
+ .with_operation("Builder::build")
+ .with_context("service", GOOSEFS_SCHEME)
+ })?;
+ }
+
+ if let Some(ref auth_username) = self.config.auth_username {
+ goosefs_config = goosefs_config.with_auth_username(auth_username);
+ }
+
+ // ── Step 3: validate the final merged config ──────────────────────────
+ goosefs_config.validate().map_err(|e| {
+ Error::new(
+ ErrorKind::ConfigInvalid,
+ format!("invalid goosefs config: {e}"),
+ )
+ .with_operation("Builder::build")
+ .with_context("service", GOOSEFS_SCHEME)
+ })?;
+
+ Ok(GooseFsBackend {
+ core: Arc::new(GooseFsCore::new(
+ {
+ let am = AccessorInfo::default();
+ am.set_scheme(GOOSEFS_SCHEME)
+ .set_root(&root)
+ .set_native_capability(Capability {
+ stat: true,
+ read: true,
+ write: true,
+ write_can_multi: true,
+ // GooseFS createFile fails with AlreadyExists if file exists,
+ // which naturally provides if_not_exists semantics.
+ // Lance Dataset relies on this for manifest commit safety.
+ write_with_if_not_exists: true,
+ create_dir: true,
+ delete: true,
+ list: true,
+ rename: true,
+ shared: true,
+ ..Default::default()
+ });
+ am.into()
+ },
+ root,
+ goosefs_config,
+ )),
+ })
+ }
+}
+
+#[derive(Debug, Clone)]
+pub struct GooseFsBackend {
Review Comment:
> Please use `Goosefs` to make the naming.
change to `GooseFS`？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/goosefs): implement opendal-service-goosefs crate backed by goosefs-sdk [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #7428:
URL: https://github.com/apache/opendal/pull/7428#issuecomment-4364139919
How to test it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/goosefs): implement opendal-service-goosefs crate backed by goosefs-sdk [opendal]
Posted by "XuQianJin-Stars (via GitHub)" <gi...@apache.org>.
XuQianJin-Stars commented on PR #7428:
URL: https://github.com/apache/opendal/pull/7428#issuecomment-4396840464
hi @Xuanwo i already updated the pr. Please help review when you got some time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/goosefs): implement opendal-service-goosefs crate backed by goosefs-sdk [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #7428:
URL: https://github.com/apache/opendal/pull/7428#discussion_r3201266346
##########
core/services/goosefs/src/backend.rs:
##########
@@ -0,0 +1,388 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+use std::sync::Arc;
+
+use log::debug;
+
+use super::GOOSEFS_SCHEME;
+use super::config::GooseFsConfig;
+use super::core::GooseFsCore;
+use super::deleter::GooseFsDeleter;
+use super::lister::GooseFsLister;
+use super::reader::GooseFsReader;
+use super::writer::GooseFsWriter;
+use super::writer::GooseFsWriters;
+use opendal_core::raw::*;
+use opendal_core::*;
+
+/// [GooseFS](https://cloud.tencent.com/product/goosefs) services support via native gRPC.
+#[doc = include_str!("docs.md")]
+#[derive(Default)]
+pub struct GooseFsBuilder {
+ pub(super) config: GooseFsConfig,
+}
+
+impl Debug for GooseFsBuilder {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("GooseFsBuilder")
+ .field("config", &self.config)
+ .finish_non_exhaustive()
+ }
+}
+
+impl GooseFsBuilder {
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(mut self, root: &str) -> Self {
+ self.config.root = if root.is_empty() {
+ None
+ } else {
+ Some(root.to_string())
+ };
+ self
+ }
+
+ /// Set master address(es).
+ ///
+ /// Single master: `"10.0.0.1:9200"`
+ /// HA (comma-separated): `"10.0.0.1:9200,10.0.0.2:9200,10.0.0.3:9200"`
+ ///
+ /// When provided here it **overrides** any address discovered from
+ /// `goosefs-site.properties` or `GOOSEFS_MASTER_ADDR`. When left unset
+ /// the builder falls back to the SDK's auto-discovery chain
+ /// (defaults → properties → env) — see `goosefs-sdk`
+ /// `docs/CLIENT_CONFIGURATION.md` §1. `build()` fails with
+ /// `ConfigInvalid` only if **no** source supplies a master address.
+ pub fn master_addr(mut self, addr: &str) -> Self {
+ if !addr.is_empty() {
+ self.config.master_addr = Some(addr.to_string());
+ }
+ self
+ }
+
+ /// Set block size for new files (bytes).
+ pub fn block_size(mut self, size: u64) -> Self {
+ self.config.block_size = Some(size);
+ self
+ }
+
+ /// Set chunk size for streaming RPCs (bytes).
+ pub fn chunk_size(mut self, size: u64) -> Self {
+ self.config.chunk_size = Some(size);
+ self
+ }
+
+ /// Set default write type.
+ ///
+ /// Values: `"must_cache"`, `"cache_through"`, `"through"`, `"async_through"`
+ pub fn write_type(mut self, wt: &str) -> Self {
+ if !wt.is_empty() {
+ self.config.write_type = Some(wt.to_string());
+ }
+ self
+ }
+
+ /// Set authentication type.
+ ///
+ /// Values: `"nosasl"`, `"simple"` (default: `"simple"`).
+ /// - `"nosasl"` — skip SASL authentication entirely.
+ /// - `"simple"` — PLAIN SASL with username (server does not verify password).
+ pub fn auth_type(mut self, auth_type: &str) -> Self {
+ if !auth_type.is_empty() {
+ self.config.auth_type = Some(auth_type.to_string());
+ }
+ self
+ }
+
+ /// Set authentication username.
+ ///
+ /// Used in SIMPLE mode as the login identity.
+ /// Default: current OS user (`$USER` / `$USERNAME`).
+ pub fn auth_username(mut self, username: &str) -> Self {
+ if !username.is_empty() {
+ self.config.auth_username = Some(username.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for GooseFsBuilder {
+ type Config = GooseFsConfig;
+
+ /// Build the backend and return a GooseFsBackend.
+ fn build(self) -> Result<impl Access> {
+ debug!("GooseFsBuilder::build started: {:?}", &self);
+
+ let root = normalize_root(&self.config.root.clone().unwrap_or_default());
+ debug!("GooseFsBuilder use root {}", &root);
+
+ // ── Step 1: establish the base SDK config ─────────────────────────────
+ //
+ // We follow the same priority chain that `FileSystemContext::connect`
+ // (and its `ConfigRefresher`) uses — see
+ // `docs/CLIENT_CONFIGURATION.md` §1 "Configuration Loading Priority":
+ //
+ // defaults < goosefs-site.properties < GOOSEFS_* env vars
+ //
+ // `GooseFsConfig::from_properties_auto()` already implements this
+ // chain and is the *same* function the SDK calls every 60s to refresh
+ // the transparent-acceleration switches. Using it here keeps the
+ // initial OpenDAL build and the in-process hot-reload semantically
+ // aligned — users who deploy `goosefs-site.properties` get the exact
+ // same config from both paths.
+ //
+ // Failure policy:
+ // * no properties file found → silently uses defaults + env
+ // (`from_properties_auto` handles this internally)
+ // * properties file present but malformed → hard-fail
+ // (broken config must not be silently dropped)
+ //
+ // Builder-explicit fields (Step 2) have the final say, overriding
+ // anything discovered from properties / env.
+ let mut goosefs_config = goosefs_sdk::config::GooseFsConfig::from_properties_auto()
+ .map_err(|e| {
+ Error::new(
+ ErrorKind::ConfigInvalid,
+ format!("failed to auto-load goosefs config: {e}"),
+ )
+ .with_operation("Builder::build")
+ .with_context("service", GOOSEFS_SCHEME)
+ })?;
+
+ // Root always comes from OpenDAL (it's an OpenDAL-layer concept).
+ goosefs_config.root = root.clone();
+
+ // ── Step 2: overlay builder-explicit fields (authoritative) ───────────
+ if let Some(ref master_addr) = self.config.master_addr {
+ let addrs: Vec<String> = master_addr
+ .split(',')
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .collect();
+
+ if addrs.is_empty() {
+ return Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ "master_addr is empty after trimming",
+ )
+ .with_operation("Builder::build")
+ .with_context("service", GOOSEFS_SCHEME));
+ }
+
+ if addrs.len() == 1 {
+ goosefs_config.master_addr = addrs[0].clone();
+ goosefs_config.master_addrs = Vec::new();
+ } else {
+ goosefs_config.master_addr = addrs[0].clone();
+ goosefs_config.master_addrs = addrs;
+ }
+ }
+
+ // After properties/env auto-load + builder overlay, we must still
+ // have at least one usable master address. If we don't, fail fast
+ // with a message that points the user at all three sources.
+ if goosefs_config.master_addr.is_empty() && goosefs_config.master_addrs.is_empty() {
+ return Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ "master_addr is not configured: set it via GooseFsBuilder::master_addr(...), \
+ the `master_addr` config key, the GOOSEFS_MASTER_ADDR env var, \
+ or `goosefs.master.hostname`/`goosefs.master.rpc.addresses` in goosefs-site.properties",
+ )
+ .with_operation("Builder::build")
+ .with_context("service", GOOSEFS_SCHEME));
+ }
+ debug!(
+ "GooseFsBuilder use master_addr {} (addrs={:?})",
+ &goosefs_config.master_addr, &goosefs_config.master_addrs
+ );
+
+ if let Some(block_size) = self.config.block_size {
+ goosefs_config.block_size = block_size;
+ }
+ if let Some(chunk_size) = self.config.chunk_size {
+ goosefs_config.chunk_size = chunk_size;
+ }
+
+ // Parse write_type string → goosefs_sdk::WritePType i32
+ if let Some(ref wt) = self.config.write_type {
+ let wt_i32 = match wt.as_str() {
+ "must_cache" | "MUST_CACHE" => 1,
Review Comment:
please convert to lower case first
##########
core/services/goosefs/src/writer.rs:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use goosefs_sdk::io::GooseFsFileWriter as ClientWriter;
+
+use super::core::GooseFsCore;
+use super::error::parse_error;
+use opendal_core::raw::*;
+use opendal_core::*;
+
+pub type GooseFsWriters = GooseFsWriter;
+
+/// GooseFsWriter implements `oio::Write` using goosefs-sdk
+/// high-level `GooseFsFileWriter`.
+///
+/// Key differences from AlluxioWriter:
+/// - Alluxio uses stream_id based REST write: `create_file() → write(stream_id, data) → close(stream_id)`
+/// - GooseFS uses block-level gRPC streaming: `GooseFsFileWriter::create() → write() → close()`
+/// which internally handles block splitting, worker routing, and gRPC bidirectional streams.
+pub struct GooseFsWriter {
+ core: Arc<GooseFsCore>,
+ op: OpWrite,
+ path: String,
+ /// Lazily initialized GooseFsFileWriter from goosefs-sdk.
+ ///
+ /// Created on first `write()` call, closed in `close()`.
+ writer: Option<ClientWriter>,
+}
+
+impl GooseFsWriter {
+ pub fn new(core: Arc<GooseFsCore>, op: OpWrite, path: String) -> Self {
+ GooseFsWriter {
+ core,
+ op,
+ path,
+ writer: None,
+ }
+ }
+}
+
+impl oio::Write for GooseFsWriter {
+ async fn write(&mut self, bs: Buffer) -> Result<()> {
+ let writer = match &mut self.writer {
+ Some(w) => w,
+ None => {
+ // Lazy init: create GooseFsFileWriter on first write.
+ //
+ // GooseFS's `createFile` RPC refuses to create a file
+ // whose path already exists — there is no native
+ // "overwrite" flag like HDFS's `CreateFlag.OVERWRITE`.
+ // OpenDAL's default `write` contract, however, is
+ // overwrite-by-default (see
+ // `async_write::test_writer_write_with_overwrite`). We
+ // emulate that semantics by pre-deleting an existing
+ // target, *unless* the caller explicitly asked for
+ // if-not-exists semantics.
+ //
+ // When `if_not_exists` is set and the file already
+ // exists, `createFile` will surface `AlreadyExists`,
+ // which we translate to `ConditionNotMatch` below so the
+ // OpenDAL contract (see
+ // `async_write::test_write_with_if_not_exists`) is
+ // honoured.
+ if !self.op.if_not_exists() {
+ // Best-effort pre-delete: ignore the result because
+ // (a) NotFound is the common case (fresh path), and
+ // (b) permission / transient errors will be
+ // reported by the subsequent create_writer call
+ // with a more informative context. The delete is
+ // non-recursive so we never stomp on a directory
+ // with the same name — if the path is a directory,
+ // the create_writer call will fail with a clear
+ // error from GooseFS.
+ let _ = self.core.delete(&self.path).await;
+ }
+
+ let w = self.core.create_writer(&self.path).await.map_err(|e| {
+ map_already_exists_for_if_not_exists(e, self.op.if_not_exists())
Review Comment:
I expect that we don't need to check `self.op` while handling errors.
##########
core/services/goosefs/src/reader.rs:
##########
@@ -0,0 +1,93 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use super::core::GooseFsCore;
+use opendal_core::raw::*;
+use opendal_core::*;
+
+/// GooseFsReader implements `oio::Read` using goosefs-sdk
+/// high-level `GooseFsFileReader`.
+///
+/// Unlike Alluxio which returns `HttpBody` (streaming HTTP response),
+/// GooseFS uses block-level gRPC streaming. The reader lazily opens
+/// a `GooseFsFileReader` on first `read()` call.
+pub struct GooseFsReader {
+ core: Arc<GooseFsCore>,
+ path: String,
+ args: OpRead,
+ /// Cached file data — we read all requested data on first call.
+ ///
+ /// This is a simplified implementation. For very large files,
+ /// a future optimization would be to stream block-by-block.
+ data: Option<Buffer>,
+ done: bool,
+}
+
+impl GooseFsReader {
+ pub fn new(core: Arc<GooseFsCore>, path: String, args: OpRead) -> Self {
+ GooseFsReader {
+ core,
+ path,
+ args,
+ data: None,
+ done: false,
+ }
+ }
+}
+
+impl oio::Read for GooseFsReader {
+ async fn read(&mut self) -> Result<Buffer> {
+ if self.done {
+ return Ok(Buffer::new());
+ }
+
+ // If we already have cached data, return it and mark done
+ if let Some(data) = self.data.take() {
+ self.done = true;
+ return Ok(data);
+ }
+
+ // Lazy initialization: read from GooseFS on first call
+ let range = self.args.range();
+ let offset = range.offset();
+ let size = range.size();
+
+ let bytes = if offset > 0 || size.is_some() {
+ // Range read
+ let len = match size {
+ Some(s) => s,
+ None => {
+ // Need file length first
+ let info = self.core.get_status(&self.path).await?;
+ let file_len = info.length.unwrap_or(0) as u64;
+ file_len.saturating_sub(offset)
+ }
+ };
+ self.core
+ .read_file(&self.path, Some(offset), Some(len))
+ .await?
+ } else {
+ // Full file read
Review Comment:
What will happen if this file is 1TiB?
##########
core/services/goosefs/src/writer.rs:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use goosefs_sdk::io::GooseFsFileWriter as ClientWriter;
+
+use super::core::GooseFsCore;
+use super::error::parse_error;
+use opendal_core::raw::*;
+use opendal_core::*;
+
+pub type GooseFsWriters = GooseFsWriter;
+
+/// GooseFsWriter implements `oio::Write` using goosefs-sdk
+/// high-level `GooseFsFileWriter`.
+///
+/// Key differences from AlluxioWriter:
+/// - Alluxio uses stream_id based REST write: `create_file() → write(stream_id, data) → close(stream_id)`
+/// - GooseFS uses block-level gRPC streaming: `GooseFsFileWriter::create() → write() → close()`
+/// which internally handles block splitting, worker routing, and gRPC bidirectional streams.
+pub struct GooseFsWriter {
+ core: Arc<GooseFsCore>,
+ op: OpWrite,
+ path: String,
+ /// Lazily initialized GooseFsFileWriter from goosefs-sdk.
+ ///
+ /// Created on first `write()` call, closed in `close()`.
+ writer: Option<ClientWriter>,
+}
+
+impl GooseFsWriter {
+ pub fn new(core: Arc<GooseFsCore>, op: OpWrite, path: String) -> Self {
+ GooseFsWriter {
+ core,
+ op,
+ path,
+ writer: None,
+ }
+ }
+}
+
+impl oio::Write for GooseFsWriter {
+ async fn write(&mut self, bs: Buffer) -> Result<()> {
+ let writer = match &mut self.writer {
+ Some(w) => w,
+ None => {
+ // Lazy init: create GooseFsFileWriter on first write.
+ //
+ // GooseFS's `createFile` RPC refuses to create a file
+ // whose path already exists — there is no native
+ // "overwrite" flag like HDFS's `CreateFlag.OVERWRITE`.
+ // OpenDAL's default `write` contract, however, is
+ // overwrite-by-default (see
+ // `async_write::test_writer_write_with_overwrite`). We
+ // emulate that semantics by pre-deleting an existing
+ // target, *unless* the caller explicitly asked for
+ // if-not-exists semantics.
+ //
+ // When `if_not_exists` is set and the file already
+ // exists, `createFile` will surface `AlreadyExists`,
+ // which we translate to `ConditionNotMatch` below so the
+ // OpenDAL contract (see
+ // `async_write::test_write_with_if_not_exists`) is
+ // honoured.
+ if !self.op.if_not_exists() {
+ // Best-effort pre-delete: ignore the result because
+ // (a) NotFound is the common case (fresh path), and
+ // (b) permission / transient errors will be
+ // reported by the subsequent create_writer call
+ // with a more informative context. The delete is
+ // non-recursive so we never stomp on a directory
+ // with the same name — if the path is a directory,
+ // the create_writer call will fail with a clear
+ // error from GooseFS.
+ let _ = self.core.delete(&self.path).await;
+ }
+
+ let w = self.core.create_writer(&self.path).await.map_err(|e| {
+ map_already_exists_for_if_not_exists(e, self.op.if_not_exists())
+ })?;
+ self.writer = Some(w);
+ self.writer.as_mut().unwrap()
+ }
+ };
+
+ writer.write(&bs.to_bytes()).await.map_err(parse_error)?;
Review Comment:
`bs.to_bytes()` will concat the internally bytes into one. If possible, please write as an iterator to avoid this.
##########
core/services/goosefs/src/backend.rs:
##########
@@ -0,0 +1,388 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+use std::sync::Arc;
+
+use log::debug;
+
+use super::GOOSEFS_SCHEME;
+use super::config::GooseFsConfig;
+use super::core::GooseFsCore;
+use super::deleter::GooseFsDeleter;
+use super::lister::GooseFsLister;
+use super::reader::GooseFsReader;
+use super::writer::GooseFsWriter;
+use super::writer::GooseFsWriters;
+use opendal_core::raw::*;
+use opendal_core::*;
+
+/// [GooseFS](https://cloud.tencent.com/product/goosefs) services support via native gRPC.
+#[doc = include_str!("docs.md")]
+#[derive(Default)]
+pub struct GooseFsBuilder {
+ pub(super) config: GooseFsConfig,
+}
+
+impl Debug for GooseFsBuilder {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("GooseFsBuilder")
+ .field("config", &self.config)
+ .finish_non_exhaustive()
+ }
+}
+
+impl GooseFsBuilder {
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(mut self, root: &str) -> Self {
+ self.config.root = if root.is_empty() {
+ None
+ } else {
+ Some(root.to_string())
+ };
+ self
+ }
+
+ /// Set master address(es).
+ ///
+ /// Single master: `"10.0.0.1:9200"`
+ /// HA (comma-separated): `"10.0.0.1:9200,10.0.0.2:9200,10.0.0.3:9200"`
+ ///
+ /// When provided here it **overrides** any address discovered from
+ /// `goosefs-site.properties` or `GOOSEFS_MASTER_ADDR`. When left unset
+ /// the builder falls back to the SDK's auto-discovery chain
+ /// (defaults → properties → env) — see `goosefs-sdk`
+ /// `docs/CLIENT_CONFIGURATION.md` §1. `build()` fails with
+ /// `ConfigInvalid` only if **no** source supplies a master address.
+ pub fn master_addr(mut self, addr: &str) -> Self {
+ if !addr.is_empty() {
+ self.config.master_addr = Some(addr.to_string());
+ }
+ self
+ }
+
+ /// Set block size for new files (bytes).
+ pub fn block_size(mut self, size: u64) -> Self {
+ self.config.block_size = Some(size);
+ self
+ }
+
+ /// Set chunk size for streaming RPCs (bytes).
+ pub fn chunk_size(mut self, size: u64) -> Self {
+ self.config.chunk_size = Some(size);
+ self
+ }
+
+ /// Set default write type.
+ ///
+ /// Values: `"must_cache"`, `"cache_through"`, `"through"`, `"async_through"`
+ pub fn write_type(mut self, wt: &str) -> Self {
+ if !wt.is_empty() {
+ self.config.write_type = Some(wt.to_string());
+ }
+ self
+ }
+
+ /// Set authentication type.
+ ///
+ /// Values: `"nosasl"`, `"simple"` (default: `"simple"`).
+ /// - `"nosasl"` — skip SASL authentication entirely.
+ /// - `"simple"` — PLAIN SASL with username (server does not verify password).
+ pub fn auth_type(mut self, auth_type: &str) -> Self {
+ if !auth_type.is_empty() {
+ self.config.auth_type = Some(auth_type.to_string());
+ }
+ self
+ }
+
+ /// Set authentication username.
+ ///
+ /// Used in SIMPLE mode as the login identity.
+ /// Default: current OS user (`$USER` / `$USERNAME`).
+ pub fn auth_username(mut self, username: &str) -> Self {
+ if !username.is_empty() {
+ self.config.auth_username = Some(username.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for GooseFsBuilder {
+ type Config = GooseFsConfig;
+
+ /// Build the backend and return a GooseFsBackend.
+ fn build(self) -> Result<impl Access> {
+ debug!("GooseFsBuilder::build started: {:?}", &self);
+
+ let root = normalize_root(&self.config.root.clone().unwrap_or_default());
+ debug!("GooseFsBuilder use root {}", &root);
+
+ // ── Step 1: establish the base SDK config ─────────────────────────────
+ //
+ // We follow the same priority chain that `FileSystemContext::connect`
+ // (and its `ConfigRefresher`) uses — see
+ // `docs/CLIENT_CONFIGURATION.md` §1 "Configuration Loading Priority":
+ //
+ // defaults < goosefs-site.properties < GOOSEFS_* env vars
+ //
+ // `GooseFsConfig::from_properties_auto()` already implements this
+ // chain and is the *same* function the SDK calls every 60s to refresh
+ // the transparent-acceleration switches. Using it here keeps the
+ // initial OpenDAL build and the in-process hot-reload semantically
+ // aligned — users who deploy `goosefs-site.properties` get the exact
+ // same config from both paths.
+ //
+ // Failure policy:
+ // * no properties file found → silently uses defaults + env
+ // (`from_properties_auto` handles this internally)
+ // * properties file present but malformed → hard-fail
+ // (broken config must not be silently dropped)
+ //
+ // Builder-explicit fields (Step 2) have the final say, overriding
+ // anything discovered from properties / env.
+ let mut goosefs_config = goosefs_sdk::config::GooseFsConfig::from_properties_auto()
+ .map_err(|e| {
+ Error::new(
+ ErrorKind::ConfigInvalid,
+ format!("failed to auto-load goosefs config: {e}"),
+ )
+ .with_operation("Builder::build")
+ .with_context("service", GOOSEFS_SCHEME)
+ })?;
+
+ // Root always comes from OpenDAL (it's an OpenDAL-layer concept).
+ goosefs_config.root = root.clone();
+
+ // ── Step 2: overlay builder-explicit fields (authoritative) ───────────
+ if let Some(ref master_addr) = self.config.master_addr {
+ let addrs: Vec<String> = master_addr
+ .split(',')
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .collect();
+
+ if addrs.is_empty() {
+ return Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ "master_addr is empty after trimming",
+ )
+ .with_operation("Builder::build")
+ .with_context("service", GOOSEFS_SCHEME));
+ }
+
+ if addrs.len() == 1 {
+ goosefs_config.master_addr = addrs[0].clone();
+ goosefs_config.master_addrs = Vec::new();
+ } else {
+ goosefs_config.master_addr = addrs[0].clone();
+ goosefs_config.master_addrs = addrs;
+ }
+ }
+
+ // After properties/env auto-load + builder overlay, we must still
+ // have at least one usable master address. If we don't, fail fast
+ // with a message that points the user at all three sources.
+ if goosefs_config.master_addr.is_empty() && goosefs_config.master_addrs.is_empty() {
+ return Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ "master_addr is not configured: set it via GooseFsBuilder::master_addr(...), \
+ the `master_addr` config key, the GOOSEFS_MASTER_ADDR env var, \
+ or `goosefs.master.hostname`/`goosefs.master.rpc.addresses` in goosefs-site.properties",
+ )
+ .with_operation("Builder::build")
+ .with_context("service", GOOSEFS_SCHEME));
+ }
+ debug!(
+ "GooseFsBuilder use master_addr {} (addrs={:?})",
+ &goosefs_config.master_addr, &goosefs_config.master_addrs
+ );
+
+ if let Some(block_size) = self.config.block_size {
+ goosefs_config.block_size = block_size;
+ }
+ if let Some(chunk_size) = self.config.chunk_size {
+ goosefs_config.chunk_size = chunk_size;
+ }
+
+ // Parse write_type string → goosefs_sdk::WritePType i32
+ if let Some(ref wt) = self.config.write_type {
+ let wt_i32 = match wt.as_str() {
+ "must_cache" | "MUST_CACHE" => 1,
+ "try_cache" | "TRY_CACHE" => 2,
+ "cache_through" | "CACHE_THROUGH" => 3,
+ "through" | "THROUGH" => 4,
+ "async_through" | "ASYNC_THROUGH" => 5,
+ _ => 1, // default to MUST_CACHE
+ };
+ goosefs_config.write_type = Some(wt_i32);
+ }
+
+ // Parse auth_type string → goosefs_sdk::auth::AuthType
+ if let Some(ref auth_type_str) = self.config.auth_type {
+ goosefs_config = goosefs_config
+ .with_auth_type_str(auth_type_str)
+ .map_err(|e| {
+ Error::new(
+ ErrorKind::ConfigInvalid,
+ format!("invalid auth_type: {}", e),
+ )
+ .with_operation("Builder::build")
+ .with_context("service", GOOSEFS_SCHEME)
+ })?;
+ }
+
+ if let Some(ref auth_username) = self.config.auth_username {
+ goosefs_config = goosefs_config.with_auth_username(auth_username);
+ }
+
+ // ── Step 3: validate the final merged config ──────────────────────────
+ goosefs_config.validate().map_err(|e| {
+ Error::new(
+ ErrorKind::ConfigInvalid,
+ format!("invalid goosefs config: {e}"),
+ )
+ .with_operation("Builder::build")
+ .with_context("service", GOOSEFS_SCHEME)
+ })?;
+
+ Ok(GooseFsBackend {
+ core: Arc::new(GooseFsCore::new(
+ {
+ let am = AccessorInfo::default();
+ am.set_scheme(GOOSEFS_SCHEME)
+ .set_root(&root)
+ .set_native_capability(Capability {
+ stat: true,
+ read: true,
+ write: true,
+ write_can_multi: true,
+ // GooseFS createFile fails with AlreadyExists if file exists,
+ // which naturally provides if_not_exists semantics.
+ // Lance Dataset relies on this for manifest commit safety.
+ write_with_if_not_exists: true,
+ create_dir: true,
+ delete: true,
+ list: true,
+ rename: true,
+ shared: true,
+ ..Default::default()
+ });
+ am.into()
+ },
+ root,
+ goosefs_config,
+ )),
+ })
+ }
+}
+
+#[derive(Debug, Clone)]
+pub struct GooseFsBackend {
Review Comment:
Please use `Goosefs` to make the naming.
##########
core/services/goosefs/src/writer.rs:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use goosefs_sdk::io::GooseFsFileWriter as ClientWriter;
+
+use super::core::GooseFsCore;
+use super::error::parse_error;
+use opendal_core::raw::*;
+use opendal_core::*;
+
+pub type GooseFsWriters = GooseFsWriter;
+
+/// GooseFsWriter implements `oio::Write` using goosefs-sdk
+/// high-level `GooseFsFileWriter`.
+///
+/// Key differences from AlluxioWriter:
+/// - Alluxio uses stream_id based REST write: `create_file() → write(stream_id, data) → close(stream_id)`
+/// - GooseFS uses block-level gRPC streaming: `GooseFsFileWriter::create() → write() → close()`
+/// which internally handles block splitting, worker routing, and gRPC bidirectional streams.
+pub struct GooseFsWriter {
+ core: Arc<GooseFsCore>,
+ op: OpWrite,
+ path: String,
+ /// Lazily initialized GooseFsFileWriter from goosefs-sdk.
+ ///
+ /// Created on first `write()` call, closed in `close()`.
+ writer: Option<ClientWriter>,
+}
+
+impl GooseFsWriter {
+ pub fn new(core: Arc<GooseFsCore>, op: OpWrite, path: String) -> Self {
+ GooseFsWriter {
+ core,
+ op,
+ path,
+ writer: None,
+ }
+ }
+}
+
+impl oio::Write for GooseFsWriter {
+ async fn write(&mut self, bs: Buffer) -> Result<()> {
+ let writer = match &mut self.writer {
+ Some(w) => w,
+ None => {
+ // Lazy init: create GooseFsFileWriter on first write.
+ //
+ // GooseFS's `createFile` RPC refuses to create a file
+ // whose path already exists — there is no native
+ // "overwrite" flag like HDFS's `CreateFlag.OVERWRITE`.
+ // OpenDAL's default `write` contract, however, is
+ // overwrite-by-default (see
+ // `async_write::test_writer_write_with_overwrite`). We
+ // emulate that semantics by pre-deleting an existing
+ // target, *unless* the caller explicitly asked for
+ // if-not-exists semantics.
+ //
+ // When `if_not_exists` is set and the file already
+ // exists, `createFile` will surface `AlreadyExists`,
+ // which we translate to `ConditionNotMatch` below so the
+ // OpenDAL contract (see
+ // `async_write::test_write_with_if_not_exists`) is
+ // honoured.
+ if !self.op.if_not_exists() {
+ // Best-effort pre-delete: ignore the result because
+ // (a) NotFound is the common case (fresh path), and
+ // (b) permission / transient errors will be
+ // reported by the subsequent create_writer call
+ // with a more informative context. The delete is
+ // non-recursive so we never stomp on a directory
+ // with the same name — if the path is a directory,
+ // the create_writer call will fail with a clear
+ // error from GooseFS.
+ let _ = self.core.delete(&self.path).await;
+ }
+
+ let w = self.core.create_writer(&self.path).await.map_err(|e| {
+ map_already_exists_for_if_not_exists(e, self.op.if_not_exists())
+ })?;
+ self.writer = Some(w);
+ self.writer.as_mut().unwrap()
+ }
+ };
+
+ writer.write(&bs.to_bytes()).await.map_err(parse_error)?;
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<Metadata> {
+ let Some(mut writer) = self.writer.take() else {
+ // No data was written yet. OpenDAL's contract for
+ // `write(path, "")` / `writer().close()` is to materialise
+ // an empty object, so we create and immediately close a
+ // zero-byte file here.
+ //
+ // As in the `write()` path above, we honour overwrite vs
+ // if-not-exists semantics: default writes pre-delete any
+ // existing target, if-not-exists writes let GooseFS surface
+ // AlreadyExists which we translate to ConditionNotMatch.
+ if !self.op.if_not_exists() {
Review Comment:
it's possible for users to start writing a file and cancelled. So instead of calling delete for everytime user write, I suggest that we use the same logic like fs do: write into a temp file and move here.
So we only need to check and delete once while the whole write process is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/goosefs): implement opendal-service-goosefs crate backed by goosefs-sdk [opendal]
Posted by "XuQianJin-Stars (via GitHub)" <gi...@apache.org>.
XuQianJin-Stars commented on PR #7428:
URL: https://github.com/apache/opendal/pull/7428#issuecomment-4393750639
> How to test it?
> Follow-up commit `ea9447d` adds the full testing story:
>
> - **Unit tests** (no cluster): 12 tests in `config.rs` + `error.rs` covering URI parsing, defaults, and error-kind mapping. `cargo test -p opendal-service-goosefs --lib` → 12 passed.
> - **Behavior fixture**: `fixtures/goosefs/docker-compose-goosefs.yml` with two profiles — default `local` (single container, matches CI) and opt-in `distributed` (master/worker split). Healthcheck runs `goosefs fs ls /`, so `healthy` implies read/write readiness.
> - **CI**: `.github/services/goosefs/goosefs/action.yml` is auto-picked up by `plan.py::provided_cases()` — no workflow YAML changes needed. Verified locally that the matrix now includes `{service=goosefs, setup=goosefs, feature=services-goosefs}`.
> - **Image**: `docker.io/forwardxu/goosefs:2.1.0`, public (no auth on CI) and pin-tagged for reproducibility. Built from the official GooseFS 2.1.0 release tarball. Happy to mirror it to an Apache-owned registry if preferred.
> - **Docs**: new `## Testing` section in `docs.md` with copy-pasteable commands; `.env.example` updated.
>
> Local checks: `fmt` ✅ · `clippy -D warnings` ✅ · unit tests ✅ · compose `up --wait` → healthy ✅.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/goosefs): implement opendal-service-goosefs crate backed by goosefs-sdk [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #7428:
URL: https://github.com/apache/opendal/pull/7428
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org