You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by xu...@apache.org on 2026/04/08 09:59:43 UTC
(opendal) branch main updated: feat(hf): support writing and reading from both http and xet (#7185)
This is an automated email from the ASF dual-hosted git repository.
xuanwo pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opendal.git
The following commit(s) were added to refs/heads/main by this push:
new 8d3dbcc3e feat(hf): support writing and reading from both http and xet (#7185)
8d3dbcc3e is described below
commit 8d3dbcc3ef6a38a3ed2a6610cf826ef51dbe9013
Author: Krisztián Szűcs <sz...@gmail.com>
AuthorDate: Wed Apr 8 11:59:37 2026 +0200
feat(hf): support writing and reading from both http and xet (#7185)
* feat(hf): support writing and reading from both http and xet
* style(hf): cargo and taplo fmt
* build(hf): switch to use github branch as xet-core optional dependency
* fix(hf): support config from options
* fix(hf): only enable write capability if token is provided
* feat(hf): add support for deleting files
* feat(hf): implement BatchDelete instead of OneShotDelete
* feat(hf): retry requests by default
* feat(hf): support multiple upload modes and more thorough testing
* style(hf): run taplo format
* chore(deps): remove tokio dependency and use backon for exponential backoff
* chore(deps): align ctor version in xet
* chore(deps): remove not unused tempfile dependency
* refactor(hf): use a more recent streaming xet client
* chore(hf): remove duplicate deleter tests
* chore(hf): remove essentially unused size field from HfWriter
* chore(hf): update XetClient.write() call after removing progress updater arguments
* feat(hf): add support for buckets repository type
* build(hf): switch to depend on subxet - a tree-shaken single crate version of xet-core
* chore(hf): format cargo toml
* chore(hf): adjustments after rebase
* refactor(hf): make xet a mandatory dependency
* refactor(hf): remove the option to disable xet in runtime
* chore(hf): more explicit error handling
* chore(hf): remove redundant retry logic
* refactor(services/hf): migrate from subxet to xet-core data/cas_client APIs
Replace subxet dependency with direct xet-core crate references (cas_client,
data, file_reconstruction, utils). Switch from XetClient to session-based
FileUploadSession/FileDownloadSession with OnceCell lazy initialization.
* chore(hf): upgrade xet dependency to consolidated xet-data/xet-client crates
* chore(hf): switch xet deps to git ref with range-stream support
* refactor(hf): cache TranslatorConfig instead of sessions, serialize integration tests
- Cache Arc<TranslatorConfig> per read/write direction; create a fresh
FileUploadSession/FileDownloadSession per operation so that finalize()
on one write doesn't poison the next
- Factor repeated config construction into xet_config(token_type)
- Mark all write/delete integration tests with #[serial] to prevent
412 "commit has happened since" races on the shared test repo
- Add serial_test dev-dependency
* chore(hf): migrate to new XetSession API from xet-core PR #747
Replace low-level xet-data/xet-client usage with the new high-level
session API from xet_pkg (hf-xet crate, lib name `xet`):
- Single hf-xet dependency replaces xet-client + xet-data
- ONE cached XetSession per HfCore (write token, serves both reads
and writes); removes XetTokenRefresher and TranslatorConfig handling
- Writer: XetUploadCommit + XetStreamUpload replace FileUploadSession
+ SingleFileCleaner; no Mutex needed since oio::Write is &mut self
- Reader: XetDownloadStream replaces DownloadStream; range converted
from BytesRange to Option<Range<u64>> for new download_stream() API
* chore(hf): upgrade xet-core to 126f30b9 with per-operation token API
Migrate to the latest xet-core API where CAS auth tokens move from
session-level to per-operation (UploadCommitBuilder/DownloadStreamGroupBuilder).
Use token refresh URLs so xet-core manages JWT lifecycle automatically.
* chore(hf): upgrade xet-core to 014ff2d7, remove XetSessionBuilder endpoint config
* chore(hf): sort Cargo.toml dependencies with taplo
* chore(hf): migrate hf-xet dependency from git to crates.io v1.5.0
* chore(hf): use crates.io hf-xet 1.5.0, clean up per-service tests, fix 404 listing
- Switch hf-xet from pinned git rev to crates.io v1.5.0
- Remove per-service integration tests that duplicate behavior tests
- Keep XET read tests (mbpp_operator) and bucket roundtrip test
- Handle HF 404 as empty listing in HfLister
- Remove serial_test dependency
* feat(hf): always use XET for writes, fix bucket support, per-session isolation
- Remove regular (base64 inline) write path; always use XET protocol
- Remove base64 dependency (no longer needed)
- Remove preupload API and determine_upload_mode
- Remove create_dir capability (HF doesn't natively support it, same as S3)
- Fix bucket tree URL: remove revision segment for buckets
- Fix bucket stat: use HEAD on resolve URL instead of paths-info API
- Fix bucket delete: use bucket_batch(DeleteFile) instead of git commit
- Fix repo_path: convert operator-relative paths to repo-absolute for
commit/delete/batch payloads (fixes behavior tests with random root)
- Create fresh XetSession per operation to avoid concurrent interference
- Make writer close() retry-safe with CloseState enum tracking progress
- Handle "Already completed" from commit.commit() gracefully on retry
* fix(hf): prefix-based listing, CAS propagation retry
- Restore prefix-based listing: when path doesn't end with '/', list
parent directory and filter results by prefix. Fixes test_list_prefix,
test_list_file_with_recursive, and test_list_dir_with_file_path.
- Add retry with exponential backoff for CAS propagation delays in
register_file(). The XET commit can complete before data is visible
to the HF API (bucket_batch or LFS commit). Retries on "not found
in Xet storage", "LFS pointer", and "Invalid file change" errors.
* fix(hf): write metadata version mismatch, increase CAS retry delays
- Don't set version on write metadata (stat doesn't return commit OID,
causing assert_metadata mismatch in test_write_returns_metadata)
- Increase CAS propagation retry delays to 1s/2s/4s/8s (15s total)
to handle slower LFS registration on datasets
* fix(hf): enable SimulateLayer create_dir via write_can_empty
- Add write_can_empty capability so SimulateLayer can implement
create_dir by writing a zero-byte file at the directory path
- Strip trailing '/' from bucket paths in register_file() since
the bucket batch API rejects paths ending with '/'
* revert(hf): remove write_can_empty and trailing slash strip
write_can_empty doesn't help HF because the SimulateLayer writes
zero-byte files at directory paths, but HF's directory concept comes
from path hierarchy, not zero-byte objects. The create_dir tests
remain unsupported (same as before write_can_empty was added).
* fix(tests): gate create_dir-dependent list tests on create_dir capability
8 behavior tests in async_list.rs call op.create_dir() without checking
the create_dir capability, causing failures on backends like HF where
directories are implicit from file paths. Add early-return guards
matching the pattern already used by other tests (test_stat_dir,
test_delete_empty_dir, test_read_with_dir_path).
Also clean up HF backend: remove .gitkeep/create_dir workarounds,
remove trailing-slash stripping from bucket_batch, remove .gitkeep
filtering from lister.
* ci(hf): rename service dir to hf, add ephemeral bucket and dataset configs
Rename .github/services/huggingface to .github/services/hf so the
test framework uses OPENDAL_HF_* env vars (matching the hf scheme).
Add two new CI test variants with ephemeral repos:
- huggingface_bucket: creates a private bucket per run, deletes after
- huggingface_dataset: creates a private dataset per run, sequential
Both use hf CLI for repo lifecycle and 1Password for credentials.
Repo names include GITHUB_RUN_ID for uniqueness.
Requires 1Password vault entries under services/hf:
- token (HF write token)
- owner (HF user or org, e.g., "opendal")
* refactor(hf): move XET session management into HfCore
- Add xet_upload_commit() and xet_download_group() to HfCore,
encapsulating session creation, token refresh URL and headers
- Restore XetSession as a field on HfCore (shared factory)
- Make xet_token_refresh_headers() private
- Simplify writer: remove CloseState enum, use Option<XetStreamUpload>
- Simplify reader: replace inline XET setup with core.xet_download_group()
- Rename upload_commit field to commit in HfWriter
* refactor(hf): rename commit_files/bucket_batch to commit_git/commit_bucket
Align naming to clearly indicate both methods do the same thing
(register/delete files) for different repo types:
- commit_git: git-based repos (model/dataset/space)
- commit_bucket: bucket repos via NDJSON batch API
* refactor(hf): clean up writer, deleter, config
Writer:
- Rename commit/stream fields to xet_commit/xet_stream for clarity
- Move XET CAS finalization into commit() method
- Rename register_file() to commit()
- Add docstrings explaining the XET-only upload flow
Deleter:
- Extract shared NotFound error handling from commit_git/commit_bucket
Config:
- Hoist test imports to module level
* refactor(hf): remove custom retry logic from writer
Remove retry_on_cas_delay helper — CAS propagation retries should be
handled by OpenDAL's RetryLayer via temporary error marking, not by
custom retry loops in the writer. Simplify commit() to call
commit_bucket/commit_git directly.
* refactor(hf): add HfCore::is_bucket() helper
Replace verbose `self.core.repo.repo_type == RepoType::Bucket` checks
with `self.core.is_bucket()` across backend, writer, reader, deleter.
Remove now-unused RepoType imports from those modules.
* refactor(hf): rename RepoType to HfRepoType, add docstrings
- Rename RepoType to HfRepoType for clarity and namespace consistency
- Import HfUri and HfRepoType at the top of core.rs
- Add docstrings to HfCore::is_bucket() and HfCore::uri()
* refactor(hf): remove HfListerWrapper, use bucket tree API natively
- Remove HfListerWrapper — bucket tree API supports recursive=false
and returns directory entries, so HierarchyLister is unnecessary
- Simplify list() to always use PageLister<HfLister>
- Pass recursive=false explicitly for buckets (API defaults to recursive)
- Move is_bucket() from HfCore to HfRepo, use repo.is_bucket() everywhere
- Default PathInfo.size to 0 (bucket directory entries omit it)
* refactor(hf): tighten visibility, simplify writer, fix error format
* fix(hf): support deletedFolders in git commits, move commit_url to HfRepo
- Add DeletedFolder type and deletedFolders field to git commit payload
matching HF's API which distinguishes file vs folder deletions
- Split directory paths (trailing /) into deletedFolders, file paths
into deletedFiles when committing to git repos
- Move git_commit_url() from HfUri to HfRepo (commit URL depends on
repo, not on a specific file path)
- Make test_remove_all_with_prefix_exists skip gracefully when the
service has real directory semantics
* fix(ci): rename CI actions, use pre-created repos, fix test probe
- Rename huggingface_bucket → hf_bucket, huggingface_dataset → hf_dataset
- Use pre-created repos with 1Password secrets (ephemeral cleanup can't
run after tests in the current CI action architecture)
- Fix test_remove_all_with_prefix_exists: use a probe write to detect
file/dir coexistence support, keep original test paths intact
1Password entries needed under services/hf:
- token, bucket_repo_id, dataset_repo_id
* chore(hf): remove unused async-trait dep, move futures to dev-deps
---
.../huggingface => hf/hf_bucket}/action.yml | 17 +-
.../huggingface => hf/hf_dataset}/action.yml | 18 +-
.../{huggingface => hf}/huggingface/action.yml | 8 +-
core/Cargo.lock | 672 +++++++++++--
core/services/hf/Cargo.toml | 8 +
core/services/hf/src/backend.rs | 237 +++--
core/services/hf/src/config.rs | 168 ++--
core/services/hf/src/core.rs | 1032 +++++++++-----------
core/services/hf/src/deleter.rs | 88 ++
core/services/hf/src/docs.md | 9 +-
core/services/hf/src/error.rs | 2 +-
core/services/hf/src/lib.rs | 4 +
core/services/hf/src/lister.rs | 261 +++--
core/services/hf/src/reader.rs | 173 ++++
core/services/hf/src/uri.rs | 590 +++++++++++
core/services/hf/src/writer.rs | 153 +++
core/tests/behavior/async_delete.rs | 17 +-
core/tests/behavior/async_list.rs | 25 +
18 files changed, 2466 insertions(+), 1016 deletions(-)
diff --git a/.github/services/huggingface/huggingface/action.yml b/.github/services/hf/hf_bucket/action.yml
similarity index 70%
copy from .github/services/huggingface/huggingface/action.yml
copy to .github/services/hf/hf_bucket/action.yml
index 2ee74d3bc..dc72a0352 100644
--- a/.github/services/huggingface/huggingface/action.yml
+++ b/.github/services/hf/hf_bucket/action.yml
@@ -15,19 +15,22 @@
# specific language governing permissions and limitations
# under the License.
-name: huggingface
-description: "Behavior test for Huggingface File System"
+name: hf_bucket
+description: "Behavior test for Huggingface Bucket (read/write)"
runs:
using: "composite"
steps:
- name: Setup
+ uses: 1password/load-secrets-action@13f58eec611f8e5db52ec16247f58c508398f3e6 # v3
+ with:
+ export-env: true
+ env:
+ OPENDAL_HF_TOKEN: op://services/hf/token
+ OPENDAL_HF_REPO_ID: op://services/hf/bucket_repo_id
+ - name: Setup env
shell: bash
run: |
cat << EOF >> $GITHUB_ENV
- OPENDAL_HUGGINGFACE_REPO_TYPE=dataset
- OPENDAL_HUGGINGFACE_REPO_ID=opendal/huggingface-testdata
- OPENDAL_HUGGINGFACE_REVISION=main
- OPENDAL_HUGGINGFACE_ROOT=/testdata/
- OPENDAL_DISABLE_RANDOM_ROOT=true
+ OPENDAL_HF_REPO_TYPE=bucket
EOF
diff --git a/.github/services/huggingface/huggingface/action.yml b/.github/services/hf/hf_dataset/action.yml
similarity index 68%
copy from .github/services/huggingface/huggingface/action.yml
copy to .github/services/hf/hf_dataset/action.yml
index 2ee74d3bc..1659bc730 100644
--- a/.github/services/huggingface/huggingface/action.yml
+++ b/.github/services/hf/hf_dataset/action.yml
@@ -15,19 +15,23 @@
# specific language governing permissions and limitations
# under the License.
-name: huggingface
-description: "Behavior test for Huggingface File System"
+name: hf_dataset
+description: "Behavior test for Huggingface Dataset (read/write)"
runs:
using: "composite"
steps:
- name: Setup
+ uses: 1password/load-secrets-action@13f58eec611f8e5db52ec16247f58c508398f3e6 # v3
+ with:
+ export-env: true
+ env:
+ OPENDAL_HF_TOKEN: op://services/hf/token
+ OPENDAL_HF_REPO_ID: op://services/hf/dataset_repo_id
+ - name: Setup env
shell: bash
run: |
cat << EOF >> $GITHUB_ENV
- OPENDAL_HUGGINGFACE_REPO_TYPE=dataset
- OPENDAL_HUGGINGFACE_REPO_ID=opendal/huggingface-testdata
- OPENDAL_HUGGINGFACE_REVISION=main
- OPENDAL_HUGGINGFACE_ROOT=/testdata/
- OPENDAL_DISABLE_RANDOM_ROOT=true
+ OPENDAL_HF_REPO_TYPE=dataset
+ RUST_TEST_THREADS=1
EOF
diff --git a/.github/services/huggingface/huggingface/action.yml b/.github/services/hf/huggingface/action.yml
similarity index 83%
rename from .github/services/huggingface/huggingface/action.yml
rename to .github/services/hf/huggingface/action.yml
index 2ee74d3bc..1174a930a 100644
--- a/.github/services/huggingface/huggingface/action.yml
+++ b/.github/services/hf/huggingface/action.yml
@@ -25,9 +25,9 @@ runs:
shell: bash
run: |
cat << EOF >> $GITHUB_ENV
- OPENDAL_HUGGINGFACE_REPO_TYPE=dataset
- OPENDAL_HUGGINGFACE_REPO_ID=opendal/huggingface-testdata
- OPENDAL_HUGGINGFACE_REVISION=main
- OPENDAL_HUGGINGFACE_ROOT=/testdata/
+ OPENDAL_HF_REPO_TYPE=dataset
+ OPENDAL_HF_REPO_ID=opendal/huggingface-testdata
+ OPENDAL_HF_REVISION=main
+ OPENDAL_HF_ROOT=/testdata/
OPENDAL_DISABLE_RANDOM_ROOT=true
EOF
diff --git a/core/Cargo.lock b/core/Cargo.lock
index 18c2011cd..c739af59d 100644
--- a/core/Cargo.lock
+++ b/core/Cargo.lock
@@ -17,7 +17,7 @@ version = "0.8.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"cipher",
"cpufeatures",
]
@@ -39,7 +39,7 @@ version = "0.8.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"getrandom 0.3.4",
"once_cell",
"version_check",
@@ -300,7 +300,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "456b8a8feb6f42d237746d4b3e9a178494627745c3c56c6ea55d92ba50d026fc"
dependencies = [
"autocfg",
- "cfg-if",
+ "cfg-if 1.0.4",
"concurrent-queue",
"futures-io",
"futures-lite",
@@ -488,7 +488,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "70af449c9a763cb655c6a1e5338b42d99c67190824ff90658c1e30be844c0775"
dependencies = [
"awaitable-error",
- "cfg-if",
+ "cfg-if 1.0.4",
]
[[package]]
@@ -1175,7 +1175,7 @@ dependencies = [
"arrayref",
"arrayvec",
"cc",
- "cfg-if",
+ "cfg-if 1.0.4",
"constant_time_eq",
"cpufeatures",
]
@@ -1287,6 +1287,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "63044e1ae8e69f3b5a92c736ca6269b8d12fa7efe39bf34ddb06d102cf0e2cab"
dependencies = [
"memchr",
+ "regex-automata",
"serde",
]
@@ -1465,6 +1466,12 @@ dependencies = [
"nom",
]
+[[package]]
+name = "cfg-if"
+version = "0.1.10"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822"
+
[[package]]
name = "cfg-if"
version = "1.0.4"
@@ -1680,7 +1687,7 @@ version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff5c12800e82a01d12046ccc29b014e1cbbb2fbe38c52534e0d40d4fc58881d5"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"cfg_aliases",
"compio-buf",
"compio-log",
@@ -1704,7 +1711,7 @@ version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c568022f90c2e2e8ea7ff4c4e8fde500753b5b9b6b6d870e25b5e656f9ea2892"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"cfg_aliases",
"compio-buf",
"compio-driver",
@@ -1742,7 +1749,7 @@ version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bffab78b8a876111ca76450912ca6a5a164b0dd93973e342c5f438a6f478c735"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"compio-buf",
"compio-driver",
"compio-io",
@@ -1762,7 +1769,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "83fd890a129a8086af857bbe18401689c130aa6ccfc7f3c029a7800f7256af3e"
dependencies = [
"async-task",
- "cfg-if",
+ "cfg-if 1.0.4",
"compio-buf",
"compio-driver",
"compio-log",
@@ -1830,6 +1837,21 @@ dependencies = [
"tiny-keccak",
]
+[[package]]
+name = "const-str"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "18f12cc9948ed9604230cdddc7c86e270f9401ccbe3c2e98a4378c5e7632212f"
+
+[[package]]
+name = "const_panic"
+version = "0.2.15"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e262cdaac42494e3ae34c43969f9cdeb7da178bdb4b66fa6a1ea2edb4c8ae652"
+dependencies = [
+ "typewit",
+]
+
[[package]]
name = "constant_time_eq"
version = "0.4.2"
@@ -1882,6 +1904,15 @@ dependencies = [
"winapi",
]
+[[package]]
+name = "countio"
+version = "0.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b9702aee5d1d744c01d82f6915644f950f898e014903385464c773b96fefdecb"
+dependencies = [
+ "futures-io",
+]
+
[[package]]
name = "cpufeatures"
version = "0.2.17"
@@ -1940,7 +1971,7 @@ version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
]
[[package]]
@@ -2089,6 +2120,27 @@ dependencies = [
"syn 2.0.117",
]
+[[package]]
+name = "csv"
+version = "1.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "52cd9d68cf7efc6ddfaaee42e7288d3a99d613d4b50f76ce9827ae0c6e14f938"
+dependencies = [
+ "csv-core",
+ "itoa",
+ "ryu",
+ "serde_core",
+]
+
+[[package]]
+name = "csv-core"
+version = "0.1.13"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "704a3c26996a80471189265814dbc2c257598b96b8a7feae2d31ace646bb9782"
+dependencies = [
+ "memchr",
+]
+
[[package]]
name = "ctor"
version = "0.6.3"
@@ -2120,7 +2172,7 @@ version = "4.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"cpufeatures",
"curve25519-dalek-derive",
"digest",
@@ -2252,7 +2304,7 @@ version = "5.5.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "978747c1d849a7d2ee5e8adc0159961c48fb7e5db2f06af6723b80123bb53856"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"hashbrown 0.14.5",
"lock_api",
"once_cell",
@@ -2265,7 +2317,7 @@ version = "6.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5041cc499144891f3790297212f32a74fb938e5136a14943f338ef9e0ae276cf"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"crossbeam-utils",
"hashbrown 0.14.5",
"lock_api",
@@ -2309,6 +2361,17 @@ dependencies = [
"powerfmt",
]
+[[package]]
+name = "derivative"
+version = "2.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fcc3dd5e9e9c0b295d6e1e4d811fb6f157d5ffd784b8d202fc62eac8035a770b"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 1.0.109",
+]
+
[[package]]
name = "derive-new"
version = "0.5.9"
@@ -2462,6 +2525,27 @@ dependencies = [
"subtle",
]
+[[package]]
+name = "dirs"
+version = "6.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c3e8aa94d75141228480295a7d0e7feb620b1a5ad9f12bc40be62411e38cce4e"
+dependencies = [
+ "dirs-sys",
+]
+
+[[package]]
+name = "dirs-sys"
+version = "0.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e01a3366d27ee9890022452ee61b2b63a67e6f13f58900b651ff5665f0bb1fab"
+dependencies = [
+ "libc",
+ "option-ext",
+ "redox_users",
+ "windows-sys 0.61.2",
+]
+
[[package]]
name = "displaydoc"
version = "0.2.5"
@@ -2479,7 +2563,7 @@ version = "0.1.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a405457ec78b8fe08b0e32b4a3570ab5dff6dd16eb9e76a5ee0a9d9cbd898933"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"clap",
"condtype",
"divan-macros",
@@ -2523,7 +2607,7 @@ version = "3.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e39034cee21a2f5bbb66ba0e3689819c4bb5d00382a282006e802a7ffa6c41d"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"libc",
"socket2 0.6.3",
"windows-sys 0.60.2",
@@ -2728,7 +2812,7 @@ version = "0.8.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "75030f3c4f45dafd7586dd6780965a8c7e8e285a5ecb86713e63a79c5b2766f3"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
]
[[package]]
@@ -2815,7 +2899,7 @@ version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "136d1b5283a1ab77bd9257427ffd09d8667ced0570b6f938942bc7568ed5b943"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"home",
"windows-sys 0.48.0",
]
@@ -2875,7 +2959,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f11da5062b3b09fc6c1ff2884ab3ec9a92677bcbe445055dcdf02e98dcc830bd"
dependencies = [
"anyhow",
- "cfg-if",
+ "cfg-if 1.0.4",
"foldhash 0.2.0",
"itoa",
"parking_lot 0.12.5",
@@ -3171,7 +3255,7 @@ checksum = "9db9c0e4648b13e9216d785b308d43751ca975301aeb83e607ec630b6f956944"
dependencies = [
"bincode",
"bytes",
- "cfg-if",
+ "cfg-if 1.0.4",
"itertools 0.14.0",
"madsim-tokio",
"mixtrics",
@@ -3473,6 +3557,15 @@ version = "1.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "312d2295c7302019c395cfb90dacd00a82a2eabd700429bba9c7a3f38dbbe11b"
+[[package]]
+name = "gearhash"
+version = "0.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c8cf82cf76cd16485e56295a1377c775ce708c9f1a0be6b029076d60a245d213"
+dependencies = [
+ "cfg-if 0.1.10",
+]
+
[[package]]
name = "generator"
version = "0.7.5"
@@ -3587,7 +3680,7 @@ version = "0.1.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fc3cb4d91f53b50155bdcfd23f6a4c39ae1969c2ae85982b135750cccaf5fce"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"libc",
"wasi 0.9.0+wasi-snapshot-preview1",
]
@@ -3598,7 +3691,7 @@ version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"js-sys",
"libc",
"wasi 0.11.1+wasi-snapshot-preview1",
@@ -3611,7 +3704,7 @@ version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"js-sys",
"libc",
"r-efi 5.3.0",
@@ -3625,11 +3718,13 @@ version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
+ "js-sys",
"libc",
"r-efi 6.0.0",
"wasip2",
"wasip3",
+ "wasm-bindgen",
]
[[package]]
@@ -3641,6 +3736,26 @@ dependencies = [
"prost 0.13.5",
]
+[[package]]
+name = "git-version"
+version = "0.3.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1ad568aa3db0fcbc81f2f116137f263d7304f512a1209b35b85150d3ef88ad19"
+dependencies = [
+ "git-version-macro",
+]
+
+[[package]]
+name = "git-version-macro"
+version = "0.3.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "53010ccb100b96a67bc32c0175f0ed1426b31b655d562898e57325f81c023ac0"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 2.0.117",
+]
+
[[package]]
name = "glob"
version = "0.3.3"
@@ -3678,7 +3793,7 @@ version = "0.10.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9efcab3c1958580ff1f25a2a41be1668f7603d849bb63af523b208a3cc1223b8"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"dashmap 6.1.0",
"futures-sink",
"futures-timer",
@@ -3761,7 +3876,7 @@ version = "2.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"crunchy",
"zerocopy",
]
@@ -3886,7 +4001,7 @@ dependencies = [
"tokio",
"url",
"uuid",
- "whoami",
+ "whoami 1.6.1",
]
[[package]]
@@ -3937,6 +4052,12 @@ dependencies = [
"http 1.4.0",
]
+[[package]]
+name = "heapify"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0049b265b7f201ca9ab25475b22b47fe444060126a51abe00f77d986fc5cc52e"
+
[[package]]
name = "heapless"
version = "0.6.1"
@@ -3990,6 +4111,28 @@ version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
+[[package]]
+name = "hf-xet"
+version = "1.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c263f8e6508e4f3a616cd580597c59862d7f44f004b5e654a1329d92637e5f53"
+dependencies = [
+ "async-trait",
+ "bytes",
+ "http 1.4.0",
+ "more-asserts",
+ "serde",
+ "thiserror 2.0.18",
+ "tokio",
+ "tokio-util",
+ "tracing",
+ "ulid",
+ "xet-client",
+ "xet-core-structures",
+ "xet-data",
+ "xet-runtime",
+]
+
[[package]]
name = "hickory-proto"
version = "0.25.2"
@@ -3997,7 +4140,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8a6fe56c0038198998a6f217ca4e7ef3a5e51f46163bd6dd60b5c71ca6c6502"
dependencies = [
"async-trait",
- "cfg-if",
+ "cfg-if 1.0.4",
"data-encoding",
"enum-as-inner",
"futures-channel",
@@ -4021,7 +4164,7 @@ version = "0.25.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc62a9a99b0bfb44d2ab95a7208ac952d31060efc16241c87eaf36406fecf87a"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"futures-util",
"hickory-proto",
"ipconfig",
@@ -4552,7 +4695,7 @@ version = "0.1.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e0242819d153cba4b4b05a5a8f2a7e9bbf97b6055b2a002b395c96b5ff3c0222"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
]
[[package]]
@@ -4572,7 +4715,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fdd7bddefd0a8833b88a4b68f90dae22c7450d11b354198baee3874fd811b344"
dependencies = [
"bitflags 2.11.0",
- "cfg-if",
+ "cfg-if 1.0.4",
"libc",
]
@@ -4723,7 +4866,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a87aa2bb7d2af34197c04845522473242e1aa17c12f4935d5856491a7fb8c97"
dependencies = [
"cesu8",
- "cfg-if",
+ "cfg-if 1.0.4",
"combine",
"jni-sys 0.3.1",
"log",
@@ -4804,6 +4947,23 @@ dependencies = [
"simple_asn1",
]
+[[package]]
+name = "konst"
+version = "0.4.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f660d5f887e3562f9ab6f4a14988795b694099d66b4f5dedc02d197ba9becb1d"
+dependencies = [
+ "const_panic",
+ "konst_proc_macros",
+ "typewit",
+]
+
+[[package]]
+name = "konst_proc_macros"
+version = "0.4.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e037a2e1d8d5fdbd49b16a4ea09d5d6401c1f29eca5ff29d03d3824dba16256a"
+
[[package]]
name = "kv-log-macro"
version = "1.0.7"
@@ -4888,7 +5048,7 @@ version = "0.8.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"windows-link 0.2.1",
]
@@ -4898,7 +5058,7 @@ version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "754ca22de805bb5744484a5b151a9e1a8e837d5dc232c2d7d8c2e3492edc8b60"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"windows-link 0.2.1",
]
@@ -5084,7 +5244,7 @@ version = "0.5.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff50ecb28bb86013e935fb6683ab1f6d3a20016f123c76fd4c27470076ac30f5"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"generator",
"scoped-tls",
"tracing",
@@ -5125,6 +5285,15 @@ dependencies = [
"libc",
]
+[[package]]
+name = "lz4_flex"
+version = "0.13.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "db9a0d582c2874f68138a16ce1867e0ffde6c0bb0a0df85e1f36d04146db488a"
+dependencies = [
+ "twox-hash",
+]
+
[[package]]
name = "mac"
version = "0.1.1"
@@ -5299,7 +5468,7 @@ version = "0.10.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"digest",
]
@@ -5588,6 +5757,12 @@ dependencies = [
"syn 2.0.117",
]
+[[package]]
+name = "more-asserts"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1fafa6961cabd9c63bcd77a45d7e3b7f3b552b70417831fb0f56db717e72407e"
+
[[package]]
name = "multimap"
version = "0.10.1"
@@ -5678,7 +5853,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "598beaf3cc6fdd9a5dfb1630c2800c7acd31df7aaf0f565796fba2b53ca1af1b"
dependencies = [
"bitflags 1.3.2",
- "cfg-if",
+ "cfg-if 1.0.4",
"libc",
"memoffset 0.7.1",
"pin-utils",
@@ -5838,6 +6013,15 @@ dependencies = [
"objc2-core-foundation",
]
+[[package]]
+name = "objc2-system-configuration"
+version = "0.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7216bd11cbda54ccabcab84d523dc93b858ec75ecfb3a7d89513fa22464da396"
+dependencies = [
+ "objc2-core-foundation",
+]
+
[[package]]
name = "object_store"
version = "0.13.2"
@@ -5880,6 +6064,12 @@ version = "1.70.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
+[[package]]
+name = "oneshot"
+version = "0.1.13"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "269bca4c2591a28585d6bf10d9ed0332b7d76900a1b02bec41bdc3a2cdcda107"
+
[[package]]
name = "oorandom"
version = "11.1.5"
@@ -6710,10 +6900,12 @@ version = "0.55.0"
dependencies = [
"bytes",
"futures",
+ "hf-xet",
"http 1.4.0",
"log",
"opendal-core",
"percent-encoding",
+ "reqwest 0.12.28",
"serde",
"serde_json",
"tokio",
@@ -6949,7 +7141,7 @@ name = "opendal-service-redb"
version = "0.55.0"
dependencies = [
"opendal-core",
- "redb",
+ "redb 2.6.3",
"serde",
"tokio",
]
@@ -7307,7 +7499,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "951c002c75e16ea2c65b8c7e4d3d51d5530d8dfa7d060b4776828c88cfb18ecf"
dependencies = [
"bitflags 2.11.0",
- "cfg-if",
+ "cfg-if 1.0.4",
"foreign-types",
"libc",
"once_cell",
@@ -7426,6 +7618,12 @@ dependencies = [
"tokio-stream",
]
+[[package]]
+name = "option-ext"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d"
+
[[package]]
name = "ordered-multimap"
version = "0.7.3"
@@ -7455,6 +7653,15 @@ dependencies = [
"windows-sys 0.61.2",
]
+[[package]]
+name = "os_str_bytes"
+version = "6.6.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e2355d85b9a3786f481747ced0e0ff2ba35213a1f9bd406ed906554d7af805a1"
+dependencies = [
+ "memchr",
+]
+
[[package]]
name = "outref"
version = "0.5.2"
@@ -7545,7 +7752,7 @@ version = "0.8.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "60a2cfe6f0ad2bfc16aefa463b497d5c7a5ecd44a23efa72aa342d90177356dc"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"instant",
"libc",
"redox_syscall 0.2.16",
@@ -7559,7 +7766,7 @@ version = "0.9.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"libc",
"redox_syscall 0.5.18",
"smallvec",
@@ -7899,7 +8106,7 @@ version = "3.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d0e4f59085d47d8241c88ead0f274e8a0cb551f3625263c05eb8dd897c34218"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"concurrent-queue",
"hermit-abi",
"pin-project-lite",
@@ -8055,7 +8262,7 @@ version = "0.13.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3d33c28a30771f7f96db69893f78b857f7450d7e0237e9c8fc6427a81bae7ed1"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"fnv",
"lazy_static",
"memchr",
@@ -8069,7 +8276,7 @@ version = "0.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3ca5326d8d0b950a9acd87e6a3f94745394f62e4dae1b1ee22b2bc0c394af43a"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"fnv",
"lazy_static",
"libc",
@@ -8579,6 +8786,15 @@ dependencies = [
"libc",
]
+[[package]]
+name = "redb"
+version = "3.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4ba239c1c1693315d3cc0e601db3b3965543afbf48c41730fdca2f069f510f4a"
+dependencies = [
+ "libc",
+]
+
[[package]]
name = "redis"
version = "1.1.0"
@@ -8590,7 +8806,7 @@ dependencies = [
"async-lock",
"backon",
"bytes",
- "cfg-if",
+ "cfg-if 1.0.4",
"combine",
"crc16",
"futures-channel",
@@ -8642,13 +8858,24 @@ dependencies = [
"bitflags 2.11.0",
]
+[[package]]
+name = "redox_users"
+version = "0.5.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a4e608c6638b9c18977b00b475ac1f28d14e84b27d8d42f70e0bf1e3dec127ac"
+dependencies = [
+ "getrandom 0.2.17",
+ "libredox",
+ "thiserror 2.0.18",
+]
+
[[package]]
name = "reflink-copy"
version = "0.1.29"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "13362233b147e57674c37b802d216b7c5e3dcccbed8967c84f0d8d223868ae27"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"libc",
"rustix 1.1.4",
"windows 0.62.2",
@@ -8934,12 +9161,56 @@ dependencies = [
"web-sys",
]
+[[package]]
+name = "reqwest-middleware"
+version = "0.5.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "199dda04a536b532d0cc04d7979e39b1c763ea749bf91507017069c00b96056f"
+dependencies = [
+ "anyhow",
+ "async-trait",
+ "http 1.4.0",
+ "reqwest 0.13.2",
+ "thiserror 2.0.18",
+ "tower-service",
+]
+
+[[package]]
+name = "reqwest-retry"
+version = "0.9.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fe2412db2af7d2268e7a5406be0431f37d9eb67ff390f35b395716f5f06c2eaa"
+dependencies = [
+ "anyhow",
+ "async-trait",
+ "futures",
+ "getrandom 0.2.17",
+ "http 1.4.0",
+ "hyper 1.8.1",
+ "reqwest 0.13.2",
+ "reqwest-middleware",
+ "retry-policies",
+ "thiserror 2.0.18",
+ "tokio",
+ "tracing",
+ "wasmtimer",
+]
+
[[package]]
name = "resolv-conf"
version = "0.7.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e061d1b48cb8d38042de4ae0a7a6401009d6143dc80d2e2d6f31f0bdd6470c7"
+[[package]]
+name = "retry-policies"
+version = "0.5.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "46a4bd6027df676bcb752d3724db0ea3c0c5fc1dd0376fec51ac7dcaf9cc69be"
+dependencies = [
+ "rand 0.9.2",
+]
+
[[package]]
name = "revision"
version = "0.17.0"
@@ -8995,7 +9266,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
dependencies = [
"cc",
- "cfg-if",
+ "cfg-if 1.0.4",
"getrandom 0.2.17",
"libc",
"untrusted 0.9.0",
@@ -9166,7 +9437,7 @@ version = "0.26.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9c845311f0ff7951c5506121a9ad75aec44d083c31583b2ea5a30bcb0b0abba0"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"glob",
"proc-macro-crate",
"proc-macro2",
@@ -9190,7 +9461,7 @@ version = "0.21.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "796e8d2b6696392a43bea58116b667fb4c29727dc5abd27d6acf338bb4f688c7"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"ordered-multimap",
]
@@ -9409,6 +9680,12 @@ version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
+[[package]]
+name = "safe-transmute"
+version = "0.11.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3944826ff8fa8093089aba3acb4ef44b9446a99a16f3bf4e74af3f77d340ab7d"
+
[[package]]
name = "salsa20"
version = "0.10.2"
@@ -9632,6 +9909,17 @@ dependencies = [
"zmij",
]
+[[package]]
+name = "serde_repr"
+version = "0.1.20"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "175ee3e80ae9982737ca543e96133087cbd9a485eecc3bc4de9c1a37b47ea59c"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 2.0.117",
+]
+
[[package]]
name = "serde_spanned"
version = "1.1.0"
@@ -9681,7 +9969,7 @@ version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f5058ada175748e33390e40e872bd0fe59a19f265d0158daa551c5a88a76009c"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"cpufeatures",
"digest",
]
@@ -9692,7 +9980,7 @@ version = "0.10.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"cpufeatures",
"digest",
]
@@ -9709,9 +9997,19 @@ version = "0.10.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"cpufeatures",
"digest",
+ "sha2-asm",
+]
+
+[[package]]
+name = "sha2-asm"
+version = "0.6.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b845214d6175804686b2bd482bcffe96651bb2d1200742b712003504a2dac1ab"
+dependencies = [
+ "cc",
]
[[package]]
@@ -9729,6 +10027,17 @@ version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "45bb67a18fa91266cc7807181f62f9178a6873bfad7dc788c42e6430db40184f"
+[[package]]
+name = "shellexpand"
+version = "3.1.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "32824fab5e16e6c4d86dc1ba84489390419a39f97699852b66480bb87d297ed8"
+dependencies = [
+ "bstr",
+ "dirs",
+ "os_str_bytes",
+]
+
[[package]]
name = "shlex"
version = "1.3.0"
@@ -10049,7 +10358,7 @@ dependencies = [
"stringprep",
"thiserror 2.0.18",
"tracing",
- "whoami",
+ "whoami 1.6.1",
]
[[package]]
@@ -10086,7 +10395,7 @@ dependencies = [
"stringprep",
"thiserror 2.0.18",
"tracing",
- "whoami",
+ "whoami 1.6.1",
]
[[package]]
@@ -10162,6 +10471,16 @@ version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"
+[[package]]
+name = "statrs"
+version = "0.18.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2a3fe7c28c6512e766b0874335db33c94ad7b8f9054228ae1c2abd47ce7d335e"
+dependencies = [
+ "approx",
+ "num-traits",
+]
+
[[package]]
name = "storekey"
version = "0.11.0"
@@ -10387,7 +10706,7 @@ dependencies = [
"subtle",
"surrealdb-protocol",
"surrealdb-types",
- "sysinfo",
+ "sysinfo 0.37.2",
"thiserror 2.0.18",
"tokio",
"tokio-util",
@@ -10603,6 +10922,20 @@ dependencies = [
"windows 0.61.3",
]
+[[package]]
+name = "sysinfo"
+version = "0.38.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "92ab6a2f8bfe508deb3c6406578252e491d299cbbf3bc0529ecc3313aee4a52f"
+dependencies = [
+ "libc",
+ "memchr",
+ "ntapi",
+ "objc2-core-foundation",
+ "objc2-io-kit",
+ "windows 0.62.2",
+]
+
[[package]]
name = "system-configuration"
version = "0.7.0"
@@ -10737,7 +11070,7 @@ version = "1.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f60246a4944f24f6e018aa17cdeffb7818b76356965d03b07d6a9886e8962185"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
]
[[package]]
@@ -10919,6 +11252,17 @@ dependencies = [
"tokio",
]
+[[package]]
+name = "tokio-retry"
+version = "0.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7f57eb36ecbe0fc510036adff84824dd3c24bb781e21bfa67b69d556aa85214f"
+dependencies = [
+ "pin-project",
+ "rand 0.8.5",
+ "tokio",
+]
+
[[package]]
name = "tokio-rustls"
version = "0.24.1"
@@ -11239,6 +11583,18 @@ dependencies = [
"tracing-core",
]
+[[package]]
+name = "tracing-appender"
+version = "0.2.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "786d480bce6247ab75f005b14ae1624ad978d3029d9113f0a22fa1ac773faeaf"
+dependencies = [
+ "crossbeam-channel",
+ "thiserror 2.0.18",
+ "time",
+ "tracing-subscriber",
+]
+
[[package]]
name = "tracing-attributes"
version = "0.1.31"
@@ -11287,6 +11643,16 @@ dependencies = [
"web-time",
]
+[[package]]
+name = "tracing-serde"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "704b1aeb7be0d0a84fc9828cae51dab5970fee5088f83d1dd7ee6f6246fc6ff1"
+dependencies = [
+ "serde",
+ "tracing-core",
+]
+
[[package]]
name = "tracing-subscriber"
version = "0.3.23"
@@ -11297,12 +11663,15 @@ dependencies = [
"nu-ansi-term",
"once_cell",
"regex-automata",
+ "serde",
+ "serde_json",
"sharded-slab",
"smallvec",
"thread_local",
"tracing",
"tracing-core",
"tracing-log",
+ "tracing-serde",
]
[[package]]
@@ -11419,6 +11788,12 @@ version = "1.19.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb"
+[[package]]
+name = "typewit"
+version = "1.14.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f8c1ae7cc0fdb8b842d65d127cb981574b0d2b249b74d1c7a2986863dc134f71"
+
[[package]]
name = "ulid"
version = "1.2.1"
@@ -11655,6 +12030,15 @@ version = "0.11.1+wasi-snapshot-preview1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
+[[package]]
+name = "wasi"
+version = "0.14.7+wasi-0.2.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "883478de20367e224c0090af9cf5f9fa85bed63a95c1abf3afc5c083ebc06e8c"
+dependencies = [
+ "wasip2",
+]
+
[[package]]
name = "wasip2"
version = "1.0.1+wasi-0.2.4"
@@ -11679,6 +12063,15 @@ version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8dad83b4f25e74f184f64c43b150b91efe7647395b42289f38e50566d82855b"
+[[package]]
+name = "wasite"
+version = "1.0.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "66fe902b4a6b8028a753d5424909b764ccf79b7a209eac9bf97e59cda9f71a42"
+dependencies = [
+ "wasi 0.14.7+wasi-0.2.4",
+]
+
[[package]]
name = "wasix"
version = "0.13.1"
@@ -11694,7 +12087,7 @@ version = "0.2.114"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6532f9a5c1ece3798cb1c2cfdba640b9b3ba884f5db45973a6f442510a87d38e"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"once_cell",
"rustversion",
"wasm-bindgen-macro",
@@ -11707,7 +12100,7 @@ version = "0.4.64"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e9c5522b3a28661442748e09d40924dfb9ca614b21c00d3fd135720e48b67db8"
dependencies = [
- "cfg-if",
+ "cfg-if 1.0.4",
"futures-util",
"js-sys",
"once_cell",
@@ -11856,6 +12249,7 @@ dependencies = [
"js-sys",
"parking_lot 0.12.5",
"pin-utils",
+ "slab",
"wasm-bindgen",
]
@@ -11931,7 +12325,20 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d4a4db5077702ca3015d3d02d74974948aba2ad9e12ab7df718ee64ccd7e97d"
dependencies = [
"libredox",
- "wasite",
+ "wasite 0.1.0",
+ "web-sys",
+]
+
+[[package]]
+name = "whoami"
+version = "2.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d6a5b12f9df4f978d2cfdb1bd3bac52433f44393342d7ee9c25f5a1c14c0f45d"
+dependencies = [
+ "libc",
+ "libredox",
+ "objc2-system-configuration",
+ "wasite 1.0.2",
"web-sys",
]
@@ -12622,6 +13029,155 @@ dependencies = [
"rustix 1.1.4",
]
+[[package]]
+name = "xet-client"
+version = "1.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4b05838f0c85303e331ae54926cf197ddf1498e8d4b97467cbb1ad0d199dd6d6"
+dependencies = [
+ "anyhow",
+ "async-trait",
+ "base64 0.22.1",
+ "bytes",
+ "clap",
+ "crc32fast",
+ "derivative",
+ "futures",
+ "http 1.4.0",
+ "hyper 1.8.1",
+ "lazy_static",
+ "more-asserts",
+ "rand 0.9.2",
+ "redb 3.1.3",
+ "reqwest 0.13.2",
+ "reqwest-middleware",
+ "reqwest-retry",
+ "serde",
+ "serde_json",
+ "serde_repr",
+ "statrs",
+ "tempfile",
+ "thiserror 2.0.18",
+ "tokio",
+ "tokio-retry",
+ "tracing",
+ "tracing-subscriber",
+ "url",
+ "urlencoding",
+ "web-time",
+ "xet-core-structures",
+ "xet-runtime",
+]
+
+[[package]]
+name = "xet-core-structures"
+version = "1.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c4c112a2b3a69b2fcccc3f69f93f059e27c9437b57733e3acc71d3af3facf464"
+dependencies = [
+ "async-trait",
+ "base64 0.22.1",
+ "blake3",
+ "bytemuck",
+ "bytes",
+ "clap",
+ "countio",
+ "csv",
+ "futures",
+ "futures-util",
+ "getrandom 0.4.2",
+ "heapify",
+ "itertools 0.14.0",
+ "lazy_static",
+ "lz4_flex",
+ "more-asserts",
+ "rand 0.9.2",
+ "regex",
+ "safe-transmute",
+ "serde",
+ "static_assertions",
+ "tempfile",
+ "thiserror 2.0.18",
+ "tokio",
+ "tokio-util",
+ "tracing",
+ "uuid",
+ "web-time",
+ "xet-runtime",
+]
+
+[[package]]
+name = "xet-data"
+version = "1.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5cf1cb808192c1bac9067022f84fd67f97e1e908ac8206caee3fb5c7146581d5"
+dependencies = [
+ "anyhow",
+ "async-trait",
+ "bytes",
+ "chrono",
+ "clap",
+ "gearhash",
+ "http 1.4.0",
+ "itertools 0.14.0",
+ "lazy_static",
+ "more-asserts",
+ "rand 0.9.2",
+ "serde",
+ "serde_json",
+ "sha2",
+ "tempfile",
+ "thiserror 2.0.18",
+ "tokio",
+ "tokio-util",
+ "tracing",
+ "ulid",
+ "url",
+ "walkdir",
+ "xet-client",
+ "xet-core-structures",
+ "xet-runtime",
+]
+
+[[package]]
+name = "xet-runtime"
+version = "1.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9dd0b331ef495dfba9ddfb9552cb00cd777712534551af11ecbd15deb9399e6b"
+dependencies = [
+ "anyhow",
+ "async-trait",
+ "bytes",
+ "chrono",
+ "colored",
+ "const-str",
+ "ctor",
+ "dirs",
+ "futures",
+ "git-version",
+ "humantime",
+ "konst",
+ "lazy_static",
+ "libc",
+ "more-asserts",
+ "oneshot",
+ "pin-project",
+ "rand 0.9.2",
+ "reqwest 0.13.2",
+ "serde",
+ "serde_json",
+ "shellexpand",
+ "sysinfo 0.38.4",
+ "thiserror 2.0.18",
+ "tokio",
+ "tokio-util",
+ "tracing",
+ "tracing-appender",
+ "tracing-subscriber",
+ "whoami 2.1.1",
+ "winapi",
+]
+
[[package]]
name = "xml-rs"
version = "0.8.28"
diff --git a/core/services/hf/Cargo.toml b/core/services/hf/Cargo.toml
index cb42c287e..c4565c130 100644
--- a/core/services/hf/Cargo.toml
+++ b/core/services/hf/Cargo.toml
@@ -32,13 +32,21 @@ all-features = true
[dependencies]
bytes = { workspace = true }
+hf-xet = "1.5.0"
http = { workspace = true }
log = { workspace = true }
opendal-core = { path = "../../core", version = "0.55.0", default-features = false }
percent-encoding = "2"
+reqwest = { version = "0.12", default-features = false, features = [
+ "rustls-tls",
+] }
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true }
[dev-dependencies]
futures = { workspace = true }
+opendal-core = { path = "../../core", version = "0.55.0", features = [
+ "reqwest-rustls-tls",
+] }
+serde_json = { workspace = true }
tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
diff --git a/core/services/hf/src/backend.rs b/core/services/hf/src/backend.rs
index 1f79dec7b..78c6418f2 100644
--- a/core/services/hf/src/backend.rs
+++ b/core/services/hf/src/backend.rs
@@ -17,17 +17,16 @@
use std::sync::Arc;
-use bytes::Buf;
-use http::Response;
-use http::StatusCode;
use log::debug;
use super::HF_SCHEME;
use super::config::HfConfig;
use super::core::HfCore;
-use super::core::HfStatus;
-use super::error::parse_error;
+use super::deleter::HfDeleter;
use super::lister::HfLister;
+use super::reader::HfReader;
+use super::uri::{HfRepo, HfRepoType};
+use super::writer::HfWriter;
use opendal_core::raw::*;
use opendal_core::*;
@@ -46,11 +45,14 @@ impl HfBuilder {
/// - dataset
/// - datasets (alias for dataset)
/// - space
+ /// - bucket
///
/// [Reference](https://huggingface.co/docs/hub/repositories)
pub fn repo_type(mut self, repo_type: &str) -> Self {
if !repo_type.is_empty() {
- self.config.repo_type = Some(repo_type.to_string());
+ if let Ok(rt) = HfRepoType::parse(repo_type) {
+ self.config.repo_type = rt;
+ }
}
self
}
@@ -123,22 +125,10 @@ impl HfBuilder {
impl Builder for HfBuilder {
type Config = HfConfig;
- /// Build an HfBackend.
fn build(self) -> Result<impl Access> {
debug!("backend build started: {:?}", &self);
- let repo_type = match self.config.repo_type.as_deref() {
- Some("model") => Ok(RepoType::Model),
- Some("dataset") | Some("datasets") => Ok(RepoType::Dataset),
- Some("space") => Ok(RepoType::Space),
- Some(repo_type) => Err(Error::new(
- ErrorKind::ConfigInvalid,
- format!("unknown repo_type: {repo_type}").as_str(),
- )
- .with_operation("Builder::build")
- .with_context("service", HF_SCHEME)),
- None => Ok(RepoType::Model),
- }?;
+ let repo_type = self.config.repo_type;
debug!("backend use repo_type: {:?}", &repo_type);
let repo_id = match &self.config.repo_id {
@@ -174,27 +164,27 @@ impl Builder for HfBuilder {
};
debug!("backend use endpoint: {}", &endpoint);
+ let info: Arc<AccessorInfo> = {
+ let am = AccessorInfo::default();
+ am.set_scheme(HF_SCHEME).set_native_capability(Capability {
+ stat: true,
+ read: true,
+ write: token.is_some(),
+ delete: token.is_some(),
+ delete_max_size: Some(100),
+ list: true,
+ list_with_recursive: true,
+ shared: true,
+ ..Default::default()
+ });
+ am.into()
+ };
+
+ let repo = HfRepo::new(repo_type, repo_id, Some(revision.clone()));
+ debug!("backend repo uri: {:?}", repo.uri(&root, ""));
+
Ok(HfBackend {
- core: Arc::new(HfCore {
- info: {
- let am = AccessorInfo::default();
- am.set_scheme(HF_SCHEME).set_native_capability(Capability {
- stat: true,
- read: true,
- list: true,
- list_with_recursive: true,
- shared: true,
- ..Default::default()
- });
- am.into()
- },
- repo_type,
- repo_id,
- revision,
- root,
- token,
- endpoint,
- }),
+ core: Arc::new(HfCore::build(info, repo, root, token, endpoint)?),
})
}
}
@@ -202,14 +192,14 @@ impl Builder for HfBuilder {
/// Backend for Hugging Face service
#[derive(Debug, Clone)]
pub struct HfBackend {
- core: Arc<HfCore>,
+ pub(crate) core: Arc<HfCore>,
}
impl Access for HfBackend {
- type Reader = HttpBody;
- type Writer = ();
+ type Reader = HfReader;
+ type Writer = HfWriter;
type Lister = oio::PageLister<HfLister>;
- type Deleter = ();
+ type Deleter = oio::BatchDeleter<HfDeleter>;
fn info(&self) -> Arc<AccessorInfo> {
self.core.info.clone()
@@ -221,80 +211,97 @@ impl Access for HfBackend {
return Ok(RpStat::new(Metadata::new(EntryMode::DIR)));
}
- let resp = self.core.hf_path_info(path).await?;
-
- let status = resp.status();
-
- match status {
- StatusCode::OK => {
- let mut meta = parse_into_metadata(path, resp.headers())?;
- let bs = resp.into_body();
-
- let decoded_response: Vec<HfStatus> =
- serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?;
-
- // NOTE: if the file is not found, the server will return 200 with an empty array
- if let Some(status) = decoded_response.first() {
- if let Some(commit_info) = status.last_commit.as_ref() {
- meta.set_last_modified(commit_info.date.parse::<Timestamp>()?);
- }
-
- meta.set_content_length(status.size);
-
- // Use LFS OID as ETag if available, otherwise use regular OID
- let etag = if let Some(lfs) = &status.lfs {
- &lfs.oid
- } else {
- &status.oid
- };
- meta.set_etag(etag);
-
- match status.type_.as_str() {
- "directory" => meta.set_mode(EntryMode::DIR),
- "file" => meta.set_mode(EntryMode::FILE),
- _ => return Err(Error::new(ErrorKind::Unexpected, "unknown status type")),
- };
- } else {
- return Err(Error::new(ErrorKind::NotFound, "path not found"));
- }
-
- Ok(RpStat::new(meta))
+ if self.core.repo.is_bucket() {
+ if path.ends_with('/') {
+ return Ok(RpStat::new(Metadata::new(EntryMode::DIR)));
}
- _ => Err(parse_error(resp)),
+ return match self.core.maybe_xet_file(path).await? {
+ Some(file_info) => {
+ let size = file_info.file_size().unwrap_or(0);
+ Ok(RpStat::new(
+ Metadata::new(EntryMode::FILE).with_content_length(size),
+ ))
+ }
+ None => Err(Error::new(ErrorKind::NotFound, "path not found")),
+ };
}
+
+ let info = self.core.path_info(path).await?;
+ Ok(RpStat::new(info.metadata()?))
}
async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> {
- let resp = self.core.hf_resolve(path, args.range(), &args).await?;
-
- let status = resp.status();
-
- match status {
- StatusCode::OK | StatusCode::PARTIAL_CONTENT => {
- Ok((RpRead::default(), resp.into_body()))
- }
- _ => {
- let (part, mut body) = resp.into_parts();
- let buf = body.to_buffer().await?;
- Err(parse_error(Response::from_parts(part, buf)))
- }
- }
+ let reader = HfReader::try_new(&self.core, path, args.range()).await?;
+ Ok((RpRead::default(), reader))
}
async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> {
- let l = HfLister::new(self.core.clone(), path.to_string(), args.recursive());
+ let lister = HfLister::new(self.core.clone(), path.to_string(), args.recursive());
+ Ok((RpList::default(), oio::PageLister::new(lister)))
+ }
- Ok((RpList::default(), oio::PageLister::new(l)))
+ async fn write(&self, path: &str, _args: OpWrite) -> Result<(RpWrite, Self::Writer)> {
+ let writer = HfWriter::try_new(self.core.clone(), path.to_string()).await?;
+ Ok((RpWrite::default(), writer))
+ }
+
+ async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> {
+ let deleter = HfDeleter::new(self.core.clone());
+ let max_batch_size = self.core.info.full_capability().delete_max_size;
+ Ok((
+ RpDelete::default(),
+ oio::BatchDeleter::new(deleter, max_batch_size),
+ ))
}
}
-/// Repository type of Hugging Face. Supports `model`, `dataset`, and `space`.
-/// [Reference](https://huggingface.co/docs/hub/repositories)
-#[derive(Debug, Clone, Copy)]
-pub enum RepoType {
- Model,
- Dataset,
- Space,
+#[cfg(test)]
+pub(super) mod test_utils {
+ use super::HfBuilder;
+ use opendal_core::Operator;
+ use opendal_core::layers::HttpClientLayer;
+ use opendal_core::raw::HttpClient;
+
+ fn finish_operator(op: Operator) -> Operator {
+ let client = HttpClient::with(reqwest::Client::new());
+ op.layer(HttpClientLayer::new(client))
+ }
+
+ pub fn gpt2_operator() -> Operator {
+ let op = Operator::new(
+ HfBuilder::default()
+ .repo_type("model")
+ .repo_id("openai-community/gpt2"),
+ )
+ .unwrap()
+ .finish();
+ finish_operator(op)
+ }
+
+ pub fn mbpp_operator() -> Operator {
+ let op = Operator::new(
+ HfBuilder::default()
+ .repo_type("dataset")
+ .repo_id("google-research-datasets/mbpp"),
+ )
+ .unwrap()
+ .finish();
+ finish_operator(op)
+ }
+
+ pub fn testing_bucket_operator() -> Operator {
+ let repo_id = std::env::var("HF_OPENDAL_BUCKET").expect("HF_OPENDAL_BUCKET must be set");
+ let token = std::env::var("HF_OPENDAL_TOKEN").expect("HF_OPENDAL_TOKEN must be set");
+ let op = Operator::new(
+ HfBuilder::default()
+ .repo_type("bucket")
+ .repo_id(&repo_id)
+ .token(&token),
+ )
+ .unwrap()
+ .finish();
+ finish_operator(op)
+ }
}
#[cfg(test)]
@@ -318,4 +325,24 @@ mod tests {
.build()
.expect("builder should accept space repo type");
}
+
+ #[test]
+ fn test_both_schemes_are_supported() {
+ use opendal_core::OperatorRegistry;
+
+ let registry = OperatorRegistry::get();
+ super::super::register_hf_service(registry);
+
+ // Test short scheme "hf"
+ let op = registry
+ .load("hf://user/repo")
+ .expect("short scheme should be registered and work");
+ assert_eq!(op.info().scheme(), "hf");
+
+ // Test long scheme "huggingface"
+ let op = registry
+ .load("huggingface://user/repo")
+ .expect("long scheme should be registered and work");
+ assert_eq!(op.info().scheme(), "hf");
+ }
}
diff --git a/core/services/hf/src/config.rs b/core/services/hf/src/config.rs
index 15fb405e4..808a88458 100644
--- a/core/services/hf/src/config.rs
+++ b/core/services/hf/src/config.rs
@@ -15,13 +15,12 @@
// specific language governing permissions and limitations
// under the License.
-use std::fmt::Debug;
-
+use super::backend::HfBuilder;
+use super::uri::HfRepoType;
+use super::uri::HfUri;
use serde::Deserialize;
use serde::Serialize;
-
-use super::HF_SCHEME;
-use super::backend::HfBuilder;
+use std::fmt::Debug;
/// Configuration for Hugging Face service support.
#[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)]
@@ -30,11 +29,8 @@ use super::backend::HfBuilder;
pub struct HfConfig {
/// Repo type of this backend. Default is model.
///
- /// Available values:
- /// - model
- /// - dataset
- /// - datasets (alias for dataset)
- pub repo_type: Option<String>,
+ /// Default is model
+ pub repo_type: HfRepoType,
/// Repo id of this backend.
///
/// This is required.
@@ -72,57 +68,54 @@ impl opendal_core::Configurator for HfConfig {
type Builder = HfBuilder;
fn from_uri(uri: &opendal_core::OperatorUri) -> opendal_core::Result<Self> {
- let mut map = uri.options().clone();
- map.retain(|_, v| !v.is_empty());
-
- if let Some(repo_type) = uri.name() {
- if !repo_type.is_empty() {
- map.insert("repo_type".to_string(), repo_type.to_string());
+ let opts = uri.options();
+
+ // Reconstruct the full path from authority (name) and root.
+ // OperatorUri splits "hf://datasets/user/repo@rev/path" into
+ // name="datasets" and root="user/repo@rev/path".
+ let mut path = String::new();
+ if let Some(name) = uri.name() {
+ if !name.is_empty() {
+ path.push_str(name);
}
}
-
- if let Some(raw_path) = uri.root() {
- let parts: Vec<_> = raw_path.split('/').filter(|s| !s.is_empty()).collect();
-
- if parts.len() >= 2 {
- map.insert("repo_id".to_string(), format!("{}/{}", parts[0], parts[1]));
-
- if parts.len() >= 3 {
- if map.contains_key("revision") {
- let root_value = parts[2..].join("/");
- if !root_value.is_empty() {
- map.insert("root".to_string(), root_value);
- }
- } else {
- map.insert("revision".to_string(), parts[2].to_string());
- if parts.len() > 3 {
- let root_value = parts[3..].join("/");
- if !root_value.is_empty() {
- map.insert("root".to_string(), root_value);
- }
- }
- }
+ if let Some(root) = uri.root() {
+ if !root.is_empty() {
+ if !path.is_empty() {
+ path.push('/');
}
- } else if parts.is_empty() {
- // no owner/repo provided, fall back to options-only
- } else {
- return Err(opendal_core::Error::new(
- opendal_core::ErrorKind::ConfigInvalid,
- "repository owner and name are required in uri path",
- )
- .with_context("service", HF_SCHEME));
+ path.push_str(root);
}
}
- if !map.contains_key("repo_id") {
- return Err(opendal_core::Error::new(
- opendal_core::ErrorKind::ConfigInvalid,
- "repo_id is required via uri path or option",
- )
- .with_context("service", HF_SCHEME));
+ if !path.is_empty() {
+ // Full URI like "hf://datasets/user/repo@rev/path"
+ let parsed = HfUri::parse(&path)?;
+ Ok(Self {
+ repo_type: parsed.repo.repo_type,
+ repo_id: Some(parsed.repo.repo_id),
+ revision: parsed.repo.revision,
+ token: opts.get("token").cloned(),
+ endpoint: opts.get("endpoint").cloned(),
+ ..Default::default()
+ })
+ } else {
+ // Bare scheme from via_iter, all config is in options.
+ let repo_type = opts
+ .get("repo_type")
+ .map(|s| HfRepoType::parse(s))
+ .transpose()?
+ .unwrap_or_default();
+ Ok(Self {
+ repo_type,
+ repo_id: opts.get("repo_id").cloned(),
+ revision: opts.get("revision").cloned(),
+ root: opts.get("root").cloned(),
+ token: opts.get("token").cloned(),
+ endpoint: opts.get("endpoint").cloned(),
+ ..Default::default()
+ })
}
-
- Self::from_iter(map)
}
fn into_builder(self) -> Self::Builder {
@@ -137,74 +130,41 @@ mod tests {
use opendal_core::OperatorUri;
#[test]
- fn from_uri_sets_repo_type_id_and_revision() {
+ fn from_uri_with_all_components() {
let uri = OperatorUri::new(
- "hf://model/opendal/sample/main/dataset",
+ "hf://datasets/username/my_dataset@dev/train/data.csv",
Vec::<(String, String)>::new(),
)
.unwrap();
let cfg = HfConfig::from_uri(&uri).unwrap();
- assert_eq!(cfg.repo_type.as_deref(), Some("model"));
- assert_eq!(cfg.repo_id.as_deref(), Some("opendal/sample"));
- assert_eq!(cfg.revision.as_deref(), Some("main"));
- assert_eq!(cfg.root.as_deref(), Some("dataset"));
- }
-
- #[test]
- fn from_uri_uses_existing_revision_and_sets_root() {
- let uri = OperatorUri::new(
- "hf://dataset/opendal/sample/data/train",
- vec![("revision".to_string(), "dev".to_string())],
- )
- .unwrap();
-
- let cfg = HfConfig::from_uri(&uri).unwrap();
- assert_eq!(cfg.repo_type.as_deref(), Some("dataset"));
- assert_eq!(cfg.repo_id.as_deref(), Some("opendal/sample"));
+ assert_eq!(cfg.repo_type, HfRepoType::Dataset);
+ assert_eq!(cfg.repo_id.as_deref(), Some("username/my_dataset"));
assert_eq!(cfg.revision.as_deref(), Some("dev"));
- assert_eq!(cfg.root.as_deref(), Some("data/train"));
+ assert!(cfg.root.is_none());
}
#[test]
- fn from_uri_allows_options_only() {
+ fn from_uri_via_iter_options() {
+ // Simulates the via_iter path: bare scheme with options map.
let uri = OperatorUri::new(
- "hf",
+ "huggingface",
vec![
- ("repo_type".to_string(), "model".to_string()),
- ("repo_id".to_string(), "opendal/sample".to_string()),
+ ("repo_type".to_string(), "dataset".to_string()),
+ (
+ "repo_id".to_string(),
+ "opendal/huggingface-testdata".to_string(),
+ ),
("revision".to_string(), "main".to_string()),
- ("root".to_string(), "".to_string()),
+ ("root".to_string(), "/testdata/".to_string()),
],
)
.unwrap();
let cfg = HfConfig::from_uri(&uri).unwrap();
- assert_eq!(cfg.repo_type.as_deref(), Some("model"));
- assert_eq!(cfg.repo_id.as_deref(), Some("opendal/sample"));
- assert_eq!(cfg.revision.as_deref(), Some("main"));
- assert!(cfg.root.is_none());
- }
-
- #[test]
- fn from_uri_requires_owner_and_repo() {
- let uri = OperatorUri::new("hf://model/opendal", Vec::<(String, String)>::new()).unwrap();
-
- assert!(HfConfig::from_uri(&uri).is_err());
- }
-
- #[test]
- fn from_uri_huggingface_alias_works() {
- let uri = OperatorUri::new(
- "huggingface://model/opendal/sample/main/dataset",
- Vec::<(String, String)>::new(),
- )
- .unwrap();
-
- let cfg = HfConfig::from_uri(&uri).unwrap();
- assert_eq!(cfg.repo_type.as_deref(), Some("model"));
- assert_eq!(cfg.repo_id.as_deref(), Some("opendal/sample"));
+ assert_eq!(cfg.repo_type, HfRepoType::Dataset);
+ assert_eq!(cfg.repo_id.as_deref(), Some("opendal/huggingface-testdata"));
assert_eq!(cfg.revision.as_deref(), Some("main"));
- assert_eq!(cfg.root.as_deref(), Some("dataset"));
+ assert_eq!(cfg.root.as_deref(), Some("/testdata/"));
}
}
diff --git a/core/services/hf/src/core.rs b/core/services/hf/src/core.rs
index 55ad405ef..5e9e1e56b 100644
--- a/core/services/hf/src/core.rs
+++ b/core/services/hf/src/core.rs
@@ -18,38 +18,161 @@
use std::fmt::Debug;
use std::sync::Arc;
+use bytes::Buf;
use bytes::Bytes;
use http::Request;
use http::Response;
use http::header;
-use percent_encoding::{NON_ALPHANUMERIC, utf8_percent_encode};
use serde::Deserialize;
-use super::backend::RepoType;
+use xet::xet_session::{
+ XetDownloadStreamGroup, XetFileInfo, XetSession, XetSessionBuilder, XetUploadCommit,
+};
+
+use super::error::parse_error;
+use super::uri::{HfRepo, HfUri};
use opendal_core::raw::*;
use opendal_core::*;
-fn percent_encode_revision(revision: &str) -> String {
- utf8_percent_encode(revision, NON_ALPHANUMERIC).to_string()
+/// API payload structures for commit operations
+#[derive(Debug, serde::Serialize)]
+pub(super) struct CommitFile {
+ pub path: String,
+ pub content: String,
+ pub encoding: String,
+}
+
+#[derive(Debug, serde::Serialize)]
+pub(super) struct LfsFile {
+ pub path: String,
+ pub oid: String,
+ pub algo: String,
+ pub size: u64,
+}
+
+#[derive(Clone, Debug, serde::Serialize)]
+pub(super) struct DeletedFile {
+ pub path: String,
}
+#[derive(Clone, Debug, serde::Serialize)]
+pub(super) struct DeletedFolder {
+ pub path: String,
+}
+
+/// Bucket batch operation payload structures
+#[derive(Debug, serde::Serialize)]
+#[serde(tag = "type", rename_all = "camelCase")]
+pub(super) enum BucketOperation {
+ #[serde(rename_all = "camelCase")]
+ AddFile { path: String, xet_hash: String },
+ #[serde(rename_all = "camelCase")]
+ #[allow(dead_code)]
+ DeleteFile { path: String },
+}
+
+#[derive(serde::Serialize)]
+pub(super) struct MixedCommitPayload {
+ pub summary: String,
+ #[serde(skip_serializing_if = "Vec::is_empty")]
+ pub files: Vec<CommitFile>,
+ #[serde(rename = "lfsFiles", skip_serializing_if = "Vec::is_empty")]
+ pub lfs_files: Vec<LfsFile>,
+ #[serde(rename = "deletedFiles", skip_serializing_if = "Vec::is_empty")]
+ pub deleted_files: Vec<DeletedFile>,
+ #[serde(rename = "deletedFolders", skip_serializing_if = "Vec::is_empty")]
+ pub deleted_folders: Vec<DeletedFolder>,
+}
+
+// API response types
+
+#[derive(serde::Deserialize, Debug)]
+pub(super) struct CommitResponse {
+ #[allow(dead_code)]
+ #[serde(rename = "commitOid")]
+ pub commit_oid: Option<String>,
+ #[allow(dead_code)]
+ #[serde(rename = "commitUrl")]
+ pub commit_url: Option<String>,
+}
+
+#[derive(Deserialize, Eq, PartialEq, Debug)]
+#[serde(rename_all = "camelCase")]
+pub(super) struct PathInfo {
+ #[serde(rename = "type")]
+ pub type_: String,
+ #[serde(default)]
+ pub oid: Option<String>,
+ #[serde(default)]
+ pub size: u64,
+ #[serde(default)]
+ pub lfs: Option<LfsInfo>,
+ pub path: String,
+ #[serde(default)]
+ pub last_commit: Option<LastCommit>,
+}
+
+impl PathInfo {
+ pub fn entry_mode(&self) -> EntryMode {
+ match self.type_.as_str() {
+ "directory" => EntryMode::DIR,
+ "file" => EntryMode::FILE,
+ _ => EntryMode::Unknown,
+ }
+ }
+
+ pub fn metadata(&self) -> Result<Metadata> {
+ let mode = self.entry_mode();
+ let mut meta = Metadata::new(mode);
+
+ if let Some(commit_info) = self.last_commit.as_ref() {
+ meta.set_last_modified(commit_info.date.parse::<Timestamp>()?);
+ }
+
+ if mode == EntryMode::FILE {
+ meta.set_content_length(self.size);
+ // For buckets, oid may be None; for regular repos, prefer lfs.oid then oid
+ if let Some(lfs) = &self.lfs {
+ meta.set_etag(&lfs.oid);
+ } else if let Some(oid) = &self.oid {
+ meta.set_etag(oid);
+ }
+ }
+
+ Ok(meta)
+ }
+}
+
+#[derive(Deserialize, Eq, PartialEq, Debug)]
+pub(super) struct LfsInfo {
+ pub oid: String,
+}
+
+#[derive(Deserialize, Eq, PartialEq, Debug)]
+pub(super) struct LastCommit {
+ pub date: String,
+}
+
+// Core HuggingFace client that manages API interactions, authentication
+// and shared logic for reader/writer/lister.
+
+#[derive(Clone)]
pub struct HfCore {
pub info: Arc<AccessorInfo>,
-
- pub repo_type: RepoType,
- pub repo_id: String,
- pub revision: String,
+ pub repo: HfRepo,
pub root: String,
pub token: Option<String>,
pub endpoint: String,
+ /// HTTP client with redirects disabled, used by XET probes to
+ /// inspect headers on 302 responses.
+ pub no_redirect_client: HttpClient,
+ pub xet_session: XetSession,
}
impl Debug for HfCore {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("HfCore")
- .field("repo_type", &self.repo_type)
- .field("repo_id", &self.repo_id)
- .field("revision", &self.revision)
+ .field("repo", &self.repo)
.field("root", &self.root)
.field("endpoint", &self.endpoint)
.finish_non_exhaustive()
@@ -57,287 +180,356 @@ impl Debug for HfCore {
}
impl HfCore {
- pub async fn hf_path_info(&self, path: &str) -> Result<Response<Buffer>> {
- let p = build_abs_path(&self.root, path)
- .trim_end_matches('/')
- .to_string();
-
- let url = match self.repo_type {
- RepoType::Model => format!(
- "{}/api/models/{}/paths-info/{}",
- &self.endpoint,
- &self.repo_id,
- percent_encode_revision(&self.revision)
- ),
- RepoType::Dataset => format!(
- "{}/api/datasets/{}/paths-info/{}",
- &self.endpoint,
- &self.repo_id,
- percent_encode_revision(&self.revision)
- ),
- RepoType::Space => format!(
- "{}/api/spaces/{}/paths-info/{}",
- &self.endpoint,
- &self.repo_id,
- percent_encode_revision(&self.revision)
- ),
- };
+ pub fn new(
+ info: Arc<AccessorInfo>,
+ repo: HfRepo,
+ root: String,
+ token: Option<String>,
+ endpoint: String,
+ no_redirect_client: HttpClient,
+ xet_session: XetSession,
+ ) -> Self {
+ Self {
+ info,
+ repo,
+ root,
+ token,
+ endpoint,
+ no_redirect_client,
+ xet_session,
+ }
+ }
+
+ /// Build HfCore with dedicated reqwest HTTP clients.
+ ///
+ /// Uses separate clients for standard and no-redirect requests to
+ /// avoid "dispatch task is gone" errors with multiple tokio runtimes.
+ pub fn build(
+ info: Arc<AccessorInfo>,
+ repo: HfRepo,
+ root: String,
+ token: Option<String>,
+ endpoint: String,
+ ) -> Result<Self> {
+ let standard = HttpClient::with(build_reqwest(reqwest::redirect::Policy::default())?);
+ let no_redirect = HttpClient::with(build_reqwest(reqwest::redirect::Policy::none())?);
+ info.update_http_client(|_| standard);
+
+ let xet_session = XetSessionBuilder::new().build().map_err(|err| {
+ Error::new(ErrorKind::Unexpected, "failed to create xet session").set_source(err)
+ })?;
+
+ Ok(Self::new(
+ info,
+ repo,
+ root,
+ token,
+ endpoint,
+ no_redirect,
+ xet_session,
+ ))
+ }
- let mut req = Request::post(&url);
- // Inject operation to the request.
- req = req.extension(Operation::Stat);
+ fn xet_token_refresh_headers(&self) -> http::HeaderMap {
+ let mut headers = http::HeaderMap::new();
if let Some(token) = &self.token {
- let auth_header_content = format_authorization_by_bearer(token)?;
- req = req.header(header::AUTHORIZATION, auth_header_content);
+ if let Ok(val) = format!("Bearer {}", token).parse() {
+ headers.insert(header::AUTHORIZATION, val);
+ }
}
+ headers
+ }
- req = req.header(header::CONTENT_TYPE, "application/x-www-form-urlencoded");
+ /// Create a new XET upload commit with token refresh configured.
+ ///
+ /// Each call creates a fresh XET session to avoid concurrent interference.
+ pub(super) async fn xet_upload_commit(&self) -> Result<XetUploadCommit> {
+ let refresh_url = self.repo.xet_token_url(&self.endpoint, "write");
+ let refresh_headers = self.xet_token_refresh_headers();
+ self.xet_session
+ .new_upload_commit()
+ .map_err(|err| {
+ Error::new(ErrorKind::Unexpected, "failed to create xet upload commit")
+ .set_source(err)
+ })?
+ .with_token_refresh_url(refresh_url, refresh_headers)
+ .build()
+ .await
+ .map_err(|err| {
+ Error::new(ErrorKind::Unexpected, "failed to build xet upload commit")
+ .set_source(err)
+ })
+ }
- let req_body = format!("paths={}&expand=True", percent_encode_path(&p));
+ /// Create a new XET download stream group with token refresh configured.
+ ///
+ /// Each call creates a fresh XET session to avoid concurrent interference.
+ pub(super) async fn xet_download_group(&self) -> Result<XetDownloadStreamGroup> {
+ let refresh_url = self.repo.xet_token_url(&self.endpoint, "read");
+ let refresh_headers = self.xet_token_refresh_headers();
+ self.xet_session
+ .new_download_stream_group()
+ .map_err(|err| {
+ Error::new(
+ ErrorKind::Unexpected,
+ "failed to create download stream group",
+ )
+ .set_source(err)
+ })?
+ .with_token_refresh_url(refresh_url, refresh_headers)
+ .build()
+ .await
+ .map_err(|err| {
+ Error::new(
+ ErrorKind::Unexpected,
+ "failed to build download stream group",
+ )
+ .set_source(err)
+ })
+ }
- let req = req
- .body(Buffer::from(Bytes::from(req_body)))
- .map_err(new_request_build_error)?;
+ /// Build an authenticated HTTP request.
+ ///
+ /// Returns `PermissionDenied` for write operations when no token
+ /// is configured.
+ pub(super) fn request(
+ &self,
+ method: http::Method,
+ url: &str,
+ op: Operation,
+ ) -> Result<http::request::Builder> {
+ let mut req = Request::builder().method(method).uri(url).extension(op);
+ match &self.token {
+ Some(token) => {
+ if let Ok(auth) = format_authorization_by_bearer(token) {
+ req = req.header(header::AUTHORIZATION, auth);
+ }
+ }
+ None if matches!(op, Operation::Write | Operation::Delete) => {
+ return Err(Error::new(
+ ErrorKind::PermissionDenied,
+ "token is required for write operations",
+ ));
+ }
+ None => {}
+ }
+ Ok(req)
+ }
- self.info.http_client().send(req).await
+ /// Build an [`HfUri`] for the given operator-relative path.
+ pub(super) fn uri(&self, path: &str) -> HfUri {
+ self.repo.uri(&self.root, path)
}
- pub async fn hf_list(
- &self,
- path: &str,
- recursive: bool,
- cursor: Option<&str>,
- ) -> Result<Response<Buffer>> {
- let p = build_abs_path(&self.root, path)
- .trim_end_matches('/')
- .to_string();
-
- let mut url = match self.repo_type {
- RepoType::Model => format!(
- "{}/api/models/{}/tree/{}/{}?expand=True",
- &self.endpoint,
- &self.repo_id,
- percent_encode_revision(&self.revision),
- percent_encode_path(&p)
- ),
- RepoType::Dataset => format!(
- "{}/api/datasets/{}/tree/{}/{}?expand=True",
- &self.endpoint,
- &self.repo_id,
- percent_encode_revision(&self.revision),
- percent_encode_path(&p)
- ),
- RepoType::Space => format!(
- "{}/api/spaces/{}/tree/{}/{}?expand=True",
- &self.endpoint,
- &self.repo_id,
- percent_encode_revision(&self.revision),
- percent_encode_path(&p)
- ),
- };
+ /// Convert an operator-relative path to a repo-absolute path
+ /// (no leading `/`) for use in commit/delete/batch payloads.
+ pub(super) fn repo_path(&self, path: &str) -> String {
+ build_abs_path(&self.root, path)
+ .trim_start_matches('/')
+ .to_string()
+ }
- if recursive {
- url.push_str("&recursive=True");
+ /// Send a request and return the successful response or a parsed error.
+ async fn send(&self, req: Request<Buffer>) -> Result<Response<Buffer>> {
+ let resp = self.info.http_client().send(req).await?;
+ if resp.status().is_success() {
+ Ok(resp)
+ } else {
+ Err(parse_error(resp))
}
+ }
- if let Some(cursor_val) = cursor {
- url.push_str(&format!("&cursor={}", cursor_val));
- }
+ /// Send a request, check for success, and deserialize the JSON response.
+ ///
+ /// Returns the response parts (status, headers, etc.) alongside the
+ /// deserialized body so callers can inspect headers when needed.
+ pub(super) async fn send_parse<T: serde::de::DeserializeOwned>(
+ &self,
+ req: Request<Buffer>,
+ ) -> Result<(http::response::Parts, T)> {
+ let (parts, body) = self.send(req).await?.into_parts();
+ let parsed = serde_json::from_reader(body.reader()).map_err(new_json_deserialize_error)?;
+ Ok((parts, parsed))
+ }
- let mut req = Request::get(&url);
- // Inject operation to the request.
- req = req.extension(Operation::List);
- if let Some(token) = &self.token {
- let auth_header_content = format_authorization_by_bearer(token)?;
- req = req.header(header::AUTHORIZATION, auth_header_content);
- }
+ pub(super) async fn path_info(&self, path: &str) -> Result<PathInfo> {
+ let uri = self.uri(path);
+ let url = uri.paths_info_url(&self.endpoint);
+ let form_body = format!("paths={}&expand=True", percent_encode_path(&uri.path));
+
+ let req = self
+ .request(http::Method::POST, &url, Operation::Stat)?
+ .header(header::CONTENT_TYPE, "application/x-www-form-urlencoded")
+ .body(Buffer::from(Bytes::from(form_body)))
+ .map_err(new_request_build_error)?;
+ let (_, mut files) = self.send_parse::<Vec<PathInfo>>(req).await?;
- let req = req.body(Buffer::new()).map_err(new_request_build_error)?;
+ // NOTE: if the file is not found, the server will return 200 with an empty array
+ if files.is_empty() {
+ return Err(Error::new(ErrorKind::NotFound, "path not found"));
+ }
- self.info.http_client().send(req).await
+ Ok(files.remove(0))
}
- pub async fn hf_list_with_url(&self, url: &str) -> Result<Response<Buffer>> {
- let mut req = Request::get(url);
- // Inject operation to the request.
- req = req.extension(Operation::List);
- if let Some(token) = &self.token {
- let auth_header_content = format_authorization_by_bearer(token)?;
- req = req.header(header::AUTHORIZATION, auth_header_content);
+ /// Issue a HEAD request and extract XET file info (hash and size).
+ ///
+ /// Returns `None` if the `X-Xet-Hash` header is absent or empty.
+ ///
+ /// Uses a dedicated no-redirect HTTP client so we can inspect
+ /// headers (e.g. `X-Xet-Hash`) on the 302 response.
+ pub(super) async fn maybe_xet_file(&self, path: &str) -> Result<Option<XetFileInfo>> {
+ let uri = self.uri(path);
+ let url = uri.resolve_url(&self.endpoint);
+
+ let req = self
+ .request(http::Method::HEAD, &url, Operation::Stat)?
+ .body(Buffer::new())
+ .map_err(new_request_build_error)?;
+
+ let resp = self.no_redirect_client.send(req).await?;
+
+ if resp.status().is_client_error() || resp.status().is_server_error() {
+ return Err(parse_error(resp));
}
- let req = req.body(Buffer::new()).map_err(new_request_build_error)?;
+ let hash = resp
+ .headers()
+ .get("X-Xet-Hash")
+ .and_then(|v| v.to_str().ok())
+ .filter(|s| !s.is_empty());
- self.info.http_client().send(req).await
+ let Some(hash) = hash else {
+ return Ok(None);
+ };
+
+ let size = resp
+ .headers()
+ .get("X-Linked-Size")
+ .or_else(|| resp.headers().get(header::CONTENT_LENGTH))
+ .and_then(|v| v.to_str().ok())
+ .and_then(|s| s.parse::<u64>().ok())
+ .unwrap_or(0);
+
+ Ok(Some(XetFileInfo::new(hash.to_string(), size)))
}
- pub async fn hf_resolve(
+ /// Commit file changes to a git-based repo (model/dataset/space).
+ ///
+ /// Counterpart of [`commit_bucket`](Self::commit_bucket) for bucket repos.
+ pub(super) async fn commit_git(
&self,
- path: &str,
- range: BytesRange,
- _args: &OpRead,
- ) -> Result<Response<HttpBody>> {
- let p = build_abs_path(&self.root, path)
- .trim_end_matches('/')
- .to_string();
-
- let url = match self.repo_type {
- RepoType::Model => format!(
- "{}/{}/resolve/{}/{}",
- &self.endpoint,
- &self.repo_id,
- percent_encode_revision(&self.revision),
- percent_encode_path(&p)
- ),
- RepoType::Dataset => format!(
- "{}/datasets/{}/resolve/{}/{}",
- &self.endpoint,
- &self.repo_id,
- percent_encode_revision(&self.revision),
- percent_encode_path(&p)
- ),
- RepoType::Space => format!(
- "{}/spaces/{}/resolve/{}/{}",
- &self.endpoint,
- &self.repo_id,
- percent_encode_revision(&self.revision),
- percent_encode_path(&p)
- ),
+ regular_files: Vec<CommitFile>,
+ lfs_files: Vec<LfsFile>,
+ deleted_files: Vec<DeletedFile>,
+ deleted_folders: Vec<DeletedFolder>,
+ ) -> Result<CommitResponse> {
+ let url = self.repo.git_commit_url(&self.endpoint);
+
+ let payload = MixedCommitPayload {
+ summary: "Commit via OpenDAL".to_string(),
+ files: regular_files,
+ lfs_files,
+ deleted_files,
+ deleted_folders,
};
- let mut req = Request::get(&url);
+ let json_body = serde_json::to_vec(&payload).map_err(new_json_serialize_error)?;
- if let Some(token) = &self.token {
- let auth_header_content = format_authorization_by_bearer(token)?;
- req = req.header(header::AUTHORIZATION, auth_header_content);
- }
-
- if !range.is_full() {
- req = req.header(header::RANGE, range.to_header());
- }
- // Inject operation to the request.
- let req = req.extension(Operation::Read);
- let req = req.body(Buffer::new()).map_err(new_request_build_error)?;
+ let req = self
+ .request(http::Method::POST, &url, Operation::Write)?
+ .header(header::CONTENT_TYPE, "application/json")
+ .header(header::CONTENT_LENGTH, json_body.len())
+ .body(Buffer::from(json_body))
+ .map_err(new_request_build_error)?;
- self.info.http_client().fetch(req).await
+ let (_, resp) = self.send_parse::<CommitResponse>(req).await?;
+ Ok(resp)
}
-}
-#[derive(Deserialize, Eq, PartialEq, Debug)]
-#[serde(rename_all = "camelCase")]
-#[allow(dead_code)]
-pub(super) struct HfStatus {
- #[serde(rename = "type")]
- pub type_: String,
- pub oid: String,
- pub size: u64,
- pub lfs: Option<HfLfs>,
- pub path: String,
- pub last_commit: Option<HfLastCommit>,
- pub security: Option<HfSecurity>,
-}
-
-#[derive(Deserialize, Eq, PartialEq, Debug)]
-#[serde(rename_all = "camelCase")]
-#[allow(dead_code)]
-pub(super) struct HfLfs {
- pub oid: String,
- pub size: u64,
- pub pointer_size: u64,
-}
+ /// Commit file changes to a bucket repo via the NDJSON batch API.
+ ///
+ /// Counterpart of [`commit_git`](Self::commit_git) for git-based repos.
+ pub(super) async fn commit_bucket(&self, operations: Vec<BucketOperation>) -> Result<()> {
+ if operations.is_empty() {
+ return Err(Error::new(
+ ErrorKind::Unexpected,
+ "no operations to perform",
+ ));
+ }
-#[derive(Deserialize, Eq, PartialEq, Debug)]
-#[serde(rename_all = "camelCase")]
-#[allow(dead_code)]
-pub(super) struct HfLastCommit {
- pub id: String,
- pub title: String,
- pub date: String,
-}
+ let url = self.repo.bucket_batch_url(&self.endpoint);
-#[derive(Deserialize, Eq, PartialEq, Debug)]
-#[serde(rename_all = "camelCase")]
-#[allow(dead_code)]
-pub(super) struct HfSecurity {
- pub blob_id: String,
- pub safe: bool,
- pub av_scan: Option<HfAvScan>,
- pub pickle_import_scan: Option<HfPickleImportScan>,
-}
-
-#[derive(Deserialize, Eq, PartialEq, Debug)]
-#[allow(dead_code)]
-#[serde(rename_all = "camelCase")]
-pub(super) struct HfAvScan {
- pub virus_found: bool,
- pub virus_names: Option<Vec<String>>,
-}
+ let mut body = String::new();
+ for op in operations {
+ let json = serde_json::to_string(&op).map_err(new_json_serialize_error)?;
+ body.push_str(&json);
+ body.push('\n');
+ }
-#[derive(Deserialize, Eq, PartialEq, Debug)]
-#[serde(rename_all = "camelCase")]
-#[allow(dead_code)]
-pub(super) struct HfPickleImportScan {
- pub highest_safety_level: String,
- pub imports: Vec<HfImport>,
-}
+ let req = self
+ .request(http::Method::POST, &url, Operation::Write)?
+ .header(header::CONTENT_TYPE, "application/x-ndjson")
+ .header(header::CONTENT_LENGTH, body.len())
+ .body(Buffer::from(Bytes::from(body)))
+ .map_err(new_request_build_error)?;
-#[derive(Deserialize, Eq, PartialEq, Debug)]
-#[allow(dead_code)]
-pub(super) struct HfImport {
- pub module: String,
- pub name: String,
- pub safety: String,
+ self.send(req).await?;
+ Ok(())
+ }
}
#[cfg(test)]
-mod tests {
- use bytes::Bytes;
+pub(crate) mod test_utils {
use http::{Request, Response, StatusCode};
use std::sync::{Arc, Mutex};
+ use super::super::uri::HfRepoType;
use super::*;
- // Mock HTTP client that captures the request URL and headers
#[derive(Clone)]
- struct MockHttpClient {
+ pub(crate) struct MockHttpClient {
url: Arc<Mutex<Option<String>>>,
- headers: Arc<Mutex<Option<http::HeaderMap>>>,
}
impl MockHttpClient {
- fn new() -> Self {
+ pub(crate) fn new() -> Self {
Self {
url: Arc::new(Mutex::new(None)),
- headers: Arc::new(Mutex::new(None)),
}
}
- fn get_captured_url(&self) -> String {
+ pub(crate) fn get_captured_url(&self) -> String {
self.url.lock().unwrap().clone().unwrap()
}
-
- fn get_captured_headers(&self) -> http::HeaderMap {
- self.headers.lock().unwrap().clone().unwrap()
- }
}
impl HttpFetch for MockHttpClient {
async fn fetch(&self, req: Request<Buffer>) -> Result<Response<HttpBody>> {
- // Capture the URL and headers
*self.url.lock().unwrap() = Some(req.uri().to_string());
- *self.headers.lock().unwrap() = Some(req.headers().clone());
- // Return a mock response with empty body
+ // Return a minimal valid JSON response for API requests
+ let body = if req.uri().to_string().contains("/paths-info/")
+ || req.uri().to_string().contains("/tree/")
+ {
+ let data =
+ Bytes::from(r#"[{"type":"file","oid":"abc123","size":100,"path":"test.txt"}]"#);
+ let size = data.len() as u64;
+ let buffer = Buffer::from(data);
+ HttpBody::new(futures::stream::iter(vec![Ok(buffer)]), Some(size))
+ } else {
+ HttpBody::new(futures::stream::empty(), Some(0))
+ };
+
Ok(Response::builder()
.status(StatusCode::OK)
- .body(HttpBody::new(futures::stream::empty(), Some(0)))
+ .body(body)
.unwrap())
}
}
- /// Utility function to create HfCore with mocked HTTP client
- fn create_test_core(
- repo_type: RepoType,
+ pub(crate) fn create_test_core(
+ repo_type: HfRepoType,
repo_id: &str,
revision: &str,
endpoint: &str,
@@ -350,29 +542,39 @@ mod tests {
.set_native_capability(Capability::default());
info.update_http_client(|_| http_client);
- let core = HfCore {
- info: Arc::new(info),
- repo_type,
- repo_id: repo_id.to_string(),
- revision: revision.to_string(),
- root: "/".to_string(),
- token: None,
- endpoint: endpoint.to_string(),
- };
+ let xet_session = XetSessionBuilder::new()
+ .build()
+ .expect("failed to create xet session");
+ let core = HfCore::new(
+ Arc::new(info),
+ HfRepo::new(repo_type, repo_id.to_string(), Some(revision.to_string())),
+ "/".to_string(),
+ None,
+ endpoint.to_string(),
+ HttpClient::with(mock_client.clone()),
+ xet_session,
+ );
(core, mock_client)
}
+}
+
+#[cfg(test)]
+mod tests {
+ use super::super::uri::HfRepoType;
+ use super::test_utils::create_test_core;
+ use super::*;
#[tokio::test]
async fn test_hf_path_info_url_model() -> Result<()> {
let (core, mock_client) = create_test_core(
- RepoType::Model,
+ HfRepoType::Model,
"test-user/test-repo",
"main",
"https://huggingface.co",
);
- core.hf_path_info("test.txt").await?;
+ core.path_info("test.txt").await?;
let url = mock_client.get_captured_url();
assert_eq!(
@@ -386,13 +588,13 @@ mod tests {
#[tokio::test]
async fn test_hf_path_info_url_dataset() -> Result<()> {
let (core, mock_client) = create_test_core(
- RepoType::Dataset,
+ HfRepoType::Dataset,
"test-org/test-dataset",
"v1.0.0",
"https://huggingface.co",
);
- core.hf_path_info("data/file.csv").await?;
+ core.path_info("data/file.csv").await?;
let url = mock_client.get_captured_url();
assert_eq!(
@@ -406,13 +608,13 @@ mod tests {
#[tokio::test]
async fn test_hf_path_info_url_custom_endpoint() -> Result<()> {
let (core, mock_client) = create_test_core(
- RepoType::Model,
+ HfRepoType::Model,
"test-org/test-dataset",
"refs/convert/parquet",
"https://custom-hf.example.com",
);
- core.hf_path_info("model.bin").await?;
+ core.path_info("model.bin").await?;
let url = mock_client.get_captured_url();
assert_eq!(
@@ -423,120 +625,16 @@ mod tests {
Ok(())
}
- #[tokio::test]
- async fn test_hf_list_url_non_recursive() -> Result<()> {
- let (core, mock_client) = create_test_core(
- RepoType::Model,
- "org/model",
- "main",
- "https://huggingface.co",
- );
-
- core.hf_list("path1", false, None).await?;
-
- let url = mock_client.get_captured_url();
- assert_eq!(
- url,
- "https://huggingface.co/api/models/org/model/tree/main/path1?expand=True"
- );
-
- Ok(())
- }
-
- #[tokio::test]
- async fn test_hf_list_url_recursive() -> Result<()> {
- let (core, mock_client) = create_test_core(
- RepoType::Model,
- "org/model",
- "main",
- "https://huggingface.co",
- );
-
- core.hf_list("path2", true, None).await?;
-
- let url = mock_client.get_captured_url();
- assert_eq!(
- url,
- "https://huggingface.co/api/models/org/model/tree/main/path2?expand=True&recursive=True"
- );
-
- Ok(())
- }
-
- #[tokio::test]
- async fn test_hf_list_url_with_cursor() -> Result<()> {
- let (core, mock_client) = create_test_core(
- RepoType::Model,
- "org/model",
- "main",
- "https://huggingface.co",
- );
-
- core.hf_list("path3", false, Some("abc123")).await?;
-
- let url = mock_client.get_captured_url();
- assert_eq!(
- url,
- "https://huggingface.co/api/models/org/model/tree/main/path3?expand=True&cursor=abc123"
- );
-
- Ok(())
- }
-
- #[tokio::test]
- async fn test_hf_resolve_url_model() -> Result<()> {
- let (core, mock_client) = create_test_core(
- RepoType::Model,
- "user/model",
- "main",
- "https://huggingface.co",
- );
-
- let args = OpRead::default();
- core.hf_resolve("config.json", BytesRange::default(), &args)
- .await?;
-
- let url = mock_client.get_captured_url();
- assert_eq!(
- url,
- "https://huggingface.co/user/model/resolve/main/config.json"
- );
-
- Ok(())
- }
-
- #[tokio::test]
- async fn test_hf_resolve_url_dataset() -> Result<()> {
- let (core, mock_client) = create_test_core(
- RepoType::Dataset,
- "org/data",
- "v1.0",
- "https://huggingface.co",
- );
-
- let args = OpRead::default();
- core.hf_resolve("train.csv", BytesRange::default(), &args)
- .await?;
-
- let url = mock_client.get_captured_url();
- assert_eq!(
- url,
- "https://huggingface.co/datasets/org/data/resolve/v1%2E0/train.csv"
- );
-
- Ok(())
- }
-
#[tokio::test]
async fn test_hf_path_info_url_space() -> Result<()> {
let (core, mock_client) = create_test_core(
- RepoType::Space,
+ HfRepoType::Space,
"test-user/test-space",
"main",
"https://huggingface.co",
);
- core.hf_path_info("app.py").await?;
+ core.path_info("app.py").await?;
let url = mock_client.get_captured_url();
assert_eq!(
@@ -546,235 +644,13 @@ mod tests {
Ok(())
}
+}
- #[tokio::test]
- async fn test_hf_list_url_space() -> Result<()> {
- let (core, mock_client) = create_test_core(
- RepoType::Space,
- "org/space",
- "main",
- "https://huggingface.co",
- );
-
- core.hf_list("static", false, None).await?;
-
- let url = mock_client.get_captured_url();
- assert_eq!(
- url,
- "https://huggingface.co/api/spaces/org/space/tree/main/static?expand=True"
- );
-
- Ok(())
- }
-
- #[tokio::test]
- async fn test_hf_resolve_url_space() -> Result<()> {
- let (core, mock_client) = create_test_core(
- RepoType::Space,
- "user/space",
- "main",
- "https://huggingface.co",
- );
-
- let args = OpRead::default();
- core.hf_resolve("README.md", BytesRange::default(), &args)
- .await?;
-
- let url = mock_client.get_captured_url();
- assert_eq!(
- url,
- "https://huggingface.co/spaces/user/space/resolve/main/README.md"
- );
-
- Ok(())
- }
-
- #[tokio::test]
- async fn test_hf_resolve_with_range() -> Result<()> {
- let (core, mock_client) = create_test_core(
- RepoType::Model,
- "user/model",
- "main",
- "https://huggingface.co",
- );
-
- let args = OpRead::default();
- let range = BytesRange::new(0, Some(1024));
- core.hf_resolve("large_file.bin", range, &args).await?;
-
- let url = mock_client.get_captured_url();
- let headers = mock_client.get_captured_headers();
- assert_eq!(
- url,
- "https://huggingface.co/user/model/resolve/main/large_file.bin"
- );
- assert_eq!(headers.get(http::header::RANGE).unwrap(), "bytes=0-1023");
-
- Ok(())
- }
-
- #[test]
- fn parse_list_response_test() -> Result<()> {
- let resp = Bytes::from(
- r#"
- [
- {
- "type": "file",
- "oid": "45fa7c3d85ee7dd4139adbc056da25ae136a65f2",
- "size": 69512435,
- "lfs": {
- "oid": "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c",
- "size": 69512435,
- "pointerSize": 133
- },
- "path": "maelstrom/lib/maelstrom.jar"
- },
- {
- "type": "directory",
- "oid": "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c",
- "size": 69512435,
- "path": "maelstrom/lib/plugins"
- }
- ]
- "#,
- );
-
- let decoded_response =
- serde_json::from_slice::<Vec<HfStatus>>(&resp).map_err(new_json_deserialize_error)?;
-
- assert_eq!(decoded_response.len(), 2);
-
- let file_entry = HfStatus {
- type_: "file".to_string(),
- oid: "45fa7c3d85ee7dd4139adbc056da25ae136a65f2".to_string(),
- size: 69512435,
- lfs: Some(HfLfs {
- oid: "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c".to_string(),
- size: 69512435,
- pointer_size: 133,
- }),
- path: "maelstrom/lib/maelstrom.jar".to_string(),
- last_commit: None,
- security: None,
- };
-
- assert_eq!(decoded_response[0], file_entry);
-
- let dir_entry = HfStatus {
- type_: "directory".to_string(),
- oid: "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c".to_string(),
- size: 69512435,
- lfs: None,
- path: "maelstrom/lib/plugins".to_string(),
- last_commit: None,
- security: None,
- };
-
- assert_eq!(decoded_response[1], dir_entry);
-
- Ok(())
- }
-
- #[test]
- fn parse_files_info_test() -> Result<()> {
- let resp = Bytes::from(
- r#"
- [
- {
- "type": "file",
- "oid": "45fa7c3d85ee7dd4139adbc056da25ae136a65f2",
- "size": 69512435,
- "lfs": {
- "oid": "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c",
- "size": 69512435,
- "pointerSize": 133
- },
- "path": "maelstrom/lib/maelstrom.jar",
- "lastCommit": {
- "id": "bc1ef030bf3743290d5e190695ab94582e51ae2f",
- "title": "Upload 141 files",
- "date": "2023-11-17T23:50:28.000Z"
- },
- "security": {
- "blobId": "45fa7c3d85ee7dd4139adbc056da25ae136a65f2",
- "name": "maelstrom/lib/maelstrom.jar",
- "safe": true,
- "avScan": {
- "virusFound": false,
- "virusNames": null
- },
- "pickleImportScan": {
- "highestSafetyLevel": "innocuous",
- "imports": [
- {"module": "torch", "name": "FloatStorage", "safety": "innocuous"},
- {"module": "collections", "name": "OrderedDict", "safety": "innocuous"},
- {"module": "torch", "name": "LongStorage", "safety": "innocuous"},
- {"module": "torch._utils", "name": "_rebuild_tensor_v2", "safety": "innocuous"}
- ]
- }
- }
- }
- ]
- "#,
- );
-
- let decoded_response =
- serde_json::from_slice::<Vec<HfStatus>>(&resp).map_err(new_json_deserialize_error)?;
-
- assert_eq!(decoded_response.len(), 1);
-
- let file_info = HfStatus {
- type_: "file".to_string(),
- oid: "45fa7c3d85ee7dd4139adbc056da25ae136a65f2".to_string(),
- size: 69512435,
- lfs: Some(HfLfs {
- oid: "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c".to_string(),
- size: 69512435,
- pointer_size: 133,
- }),
- path: "maelstrom/lib/maelstrom.jar".to_string(),
- last_commit: Some(HfLastCommit {
- id: "bc1ef030bf3743290d5e190695ab94582e51ae2f".to_string(),
- title: "Upload 141 files".to_string(),
- date: "2023-11-17T23:50:28.000Z".to_string(),
- }),
- security: Some(HfSecurity {
- blob_id: "45fa7c3d85ee7dd4139adbc056da25ae136a65f2".to_string(),
- safe: true,
- av_scan: Some(HfAvScan {
- virus_found: false,
- virus_names: None,
- }),
- pickle_import_scan: Some(HfPickleImportScan {
- highest_safety_level: "innocuous".to_string(),
- imports: vec![
- HfImport {
- module: "torch".to_string(),
- name: "FloatStorage".to_string(),
- safety: "innocuous".to_string(),
- },
- HfImport {
- module: "collections".to_string(),
- name: "OrderedDict".to_string(),
- safety: "innocuous".to_string(),
- },
- HfImport {
- module: "torch".to_string(),
- name: "LongStorage".to_string(),
- safety: "innocuous".to_string(),
- },
- HfImport {
- module: "torch._utils".to_string(),
- name: "_rebuild_tensor_v2".to_string(),
- safety: "innocuous".to_string(),
- },
- ],
- }),
- }),
- };
-
- assert_eq!(decoded_response[0], file_info);
-
- Ok(())
- }
+fn build_reqwest(policy: reqwest::redirect::Policy) -> Result<reqwest::Client> {
+ reqwest::Client::builder()
+ .redirect(policy)
+ .build()
+ .map_err(|err| {
+ Error::new(ErrorKind::Unexpected, "failed to build http client").set_source(err)
+ })
}
diff --git a/core/services/hf/src/deleter.rs b/core/services/hf/src/deleter.rs
new file mode 100644
index 000000000..09c1e9159
--- /dev/null
+++ b/core/services/hf/src/deleter.rs
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use super::core::{BucketOperation, DeletedFile, DeletedFolder, HfCore};
+use opendal_core::raw::oio::BatchDeleteResult;
+use opendal_core::raw::*;
+use opendal_core::*;
+
+pub struct HfDeleter {
+ core: Arc<HfCore>,
+}
+
+impl HfDeleter {
+ pub fn new(core: Arc<HfCore>) -> Self {
+ Self { core }
+ }
+
+ async fn delete_paths(&self, paths: Vec<String>) -> Result<()> {
+ if paths.is_empty() {
+ return Ok(());
+ }
+
+ let result = if self.core.repo.is_bucket() {
+ let ops = paths
+ .into_iter()
+ .map(|path| BucketOperation::DeleteFile { path })
+ .collect();
+ self.core.commit_bucket(ops).await.map(|_| ())
+ } else {
+ // Git commit API distinguishes file and folder deletions.
+ let mut deleted_files = Vec::new();
+ let mut deleted_folders = Vec::new();
+ for path in paths {
+ if path.ends_with('/') {
+ deleted_folders.push(DeletedFolder { path });
+ } else {
+ deleted_files.push(DeletedFile { path });
+ }
+ }
+ self.core
+ .commit_git(vec![], vec![], deleted_files, deleted_folders)
+ .await
+ .map(|_| ())
+ };
+
+ match result {
+ Ok(()) => Ok(()),
+ Err(err) if err.kind() == ErrorKind::NotFound => Ok(()),
+ Err(err) => Err(err),
+ }
+ }
+}
+
+impl oio::BatchDelete for HfDeleter {
+ async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> {
+ let repo_path = self.core.repo_path(&path);
+ self.delete_paths(vec![repo_path]).await
+ }
+
+ async fn delete_batch(&self, batch: Vec<(String, OpDelete)>) -> Result<BatchDeleteResult> {
+ let paths: Vec<String> = batch
+ .iter()
+ .map(|(path, _)| self.core.repo_path(path))
+ .collect();
+
+ self.delete_paths(paths).await?;
+ Ok(BatchDeleteResult {
+ succeeded: batch,
+ failed: vec![],
+ })
+ }
+}
diff --git a/core/services/hf/src/docs.md b/core/services/hf/src/docs.md
index c4e50fe78..f7620f0cd 100644
--- a/core/services/hf/src/docs.md
+++ b/core/services/hf/src/docs.md
@@ -1,5 +1,4 @@
This service will visit the [Hugging Face API](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api) to access the Hugging Face File System.
-Currently, we only support the `model` and `dataset` types of repositories, and operations are limited to reading and listing/stating.
Hugging Face doesn't host official HTTP API docs. Detailed HTTP request API information can be found on the [`huggingface_hub` Source Code](https://github.com/huggingface/huggingface_hub).
@@ -12,8 +11,8 @@ This service can be used to:
- [ ] create_dir
- [x] stat
- [x] read
-- [ ] write
-- [ ] delete
+- [x] write
+- [x] delete
- [x] list
- [ ] copy
- [ ] rename
@@ -21,11 +20,11 @@ This service can be used to:
## Configurations
-- `repo_type`: The type of the repository.
+- `repo_type`: The type of the repository (model, dataset, or space).
- `repo_id`: The id of the repository.
- `revision`: The revision of the repository.
- `root`: Set the work directory for backend.
-- `token`: The token for accessing the repository.
+- `token`: The token for accessing the repository. Required for write operations.
Refer to [`HfBuilder`]'s public API docs for more information.
diff --git a/core/services/hf/src/error.rs b/core/services/hf/src/error.rs
index dd2b5e16e..c0e36616d 100644
--- a/core/services/hf/src/error.rs
+++ b/core/services/hf/src/error.rs
@@ -53,7 +53,7 @@ pub(super) fn parse_error(resp: Response<Buffer>) -> Error {
};
let message = match serde_json::from_slice::<HfError>(&bs) {
- Ok(hf_error) => format!("{:?}", hf_error.error),
+ Ok(hf_error) => hf_error.error,
Err(_) => String::from_utf8_lossy(&bs).into_owned(),
};
diff --git a/core/services/hf/src/lib.rs b/core/services/hf/src/lib.rs
index e54175b46..b50185d55 100644
--- a/core/services/hf/src/lib.rs
+++ b/core/services/hf/src/lib.rs
@@ -30,8 +30,12 @@ pub fn register_hf_service(registry: &opendal_core::OperatorRegistry) {
mod backend;
mod config;
mod core;
+mod deleter;
mod error;
mod lister;
+mod reader;
+mod uri;
+mod writer;
pub use backend::HfBuilder as Hf;
pub use config::HfConfig;
diff --git a/core/services/hf/src/lister.rs b/core/services/hf/src/lister.rs
index a01f5fe9b..83404fb74 100644
--- a/core/services/hf/src/lister.rs
+++ b/core/services/hf/src/lister.rs
@@ -17,195 +17,164 @@
use std::sync::Arc;
-use bytes::Buf;
-
-use super::core::HfCore;
-use super::core::HfStatus;
-use super::error::parse_error;
+use super::core::{HfCore, PathInfo};
use opendal_core::raw::*;
use opendal_core::*;
+struct FileTree {
+ files: Vec<PathInfo>,
+ next_cursor: Option<String>,
+}
+
+/// Extract the cursor value from a Link header's "next" URL.
+fn parse_next_cursor(link_str: &str) -> Option<String> {
+ for link in link_str.split(',') {
+ if link.contains("rel=\"next\"") || link.contains("rel='next'") {
+ let (_, rest) = link.split_once('<')?;
+ let (url, _) = rest.split_once('>')?;
+ let query = url.split_once('?')?.1;
+ return query
+ .split('&')
+ .find_map(|p| p.strip_prefix("cursor="))
+ .map(|v| v.to_string());
+ }
+ }
+ None
+}
+
pub struct HfLister {
core: Arc<HfCore>,
- path: String,
+ /// The directory path to list via the tree API (always ends with `/` or is empty for root).
+ list_path: String,
+ /// When the original path didn't end with `/`, filter results to this prefix.
+ prefix: Option<String>,
recursive: bool,
}
impl HfLister {
pub fn new(core: Arc<HfCore>, path: String, recursive: bool) -> Self {
- Self {
- core,
- path,
- recursive,
+ if path.is_empty() || path.ends_with('/') {
+ Self {
+ core,
+ list_path: path,
+ prefix: None,
+ recursive,
+ }
+ } else {
+ // Prefix listing: list the parent directory and filter by prefix.
+ let parent = match path.rfind('/') {
+ Some(pos) => path[..=pos].to_string(),
+ None => String::new(),
+ };
+ Self {
+ core,
+ list_path: parent,
+ prefix: Some(path),
+ recursive,
+ }
}
}
+
+ async fn file_tree(
+ &self,
+ path: &str,
+ recursive: bool,
+ cursor: Option<&str>,
+ ) -> Result<FileTree> {
+ let uri = self.core.uri(path);
+ let url = uri.file_tree_url(&self.core.endpoint, recursive, cursor);
+
+ let req = self
+ .core
+ .request(http::Method::GET, &url, Operation::List)?
+ .body(Buffer::new())
+ .map_err(new_request_build_error)?;
+ let (parts, files) = self.core.send_parse::<Vec<PathInfo>>(req).await?;
+
+ let next_cursor = parts
+ .headers
+ .get(http::header::LINK)
+ .and_then(|v| v.to_str().ok())
+ .and_then(parse_next_cursor);
+
+ Ok(FileTree { files, next_cursor })
+ }
}
impl oio::PageList for HfLister {
async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> {
- // Use the next page URL from context if available, otherwise start from beginning
- let response = if ctx.token.is_empty() {
- self.core.hf_list(&self.path, self.recursive, None).await?
+ let cursor = if ctx.token.is_empty() {
+ None
} else {
- self.core.hf_list_with_url(&ctx.token).await?
+ Some(ctx.token.as_str())
};
- let status_code = response.status();
- if !status_code.is_success() {
- let error = parse_error(response);
- return Err(error);
- }
-
- // Parse Link header for pagination
- let next_link = parse_link_header(response.headers());
-
- let bytes = response.into_body();
- let decoded_response: Vec<HfStatus> =
- serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?;
+ let response = match self
+ .file_tree(&self.list_path, self.recursive, cursor)
+ .await
+ {
+ Ok(r) => r,
+ // HF returns 404 when a path doesn't exist; treat as empty listing.
+ Err(e) if e.kind() == ErrorKind::NotFound => {
+ ctx.done = true;
+ return Ok(());
+ }
+ Err(e) => return Err(e),
+ };
- // Only mark as done if there's no next page
- if let Some(next_url) = next_link {
- ctx.token = next_url;
+ if let Some(next_cursor) = response.next_cursor {
+ ctx.token = next_cursor;
} else {
ctx.done = true;
}
- for status in decoded_response {
- let entry_type = match status.type_.as_str() {
- "directory" => EntryMode::DIR,
- "file" => EntryMode::FILE,
- _ => EntryMode::Unknown,
+ for info in response.files {
+ let meta = info.metadata()?;
+ let path = if meta.mode() == EntryMode::DIR {
+ format!("{}/", &info.path)
+ } else {
+ info.path.clone()
};
+ let rel_path = build_rel_path(&self.core.root, &path);
- let mut meta = Metadata::new(entry_type);
-
- if let Some(commit_info) = status.last_commit.as_ref() {
- meta.set_last_modified(commit_info.date.parse::<Timestamp>()?);
+ // Filter by prefix when doing prefix-based listing.
+ if let Some(prefix) = &self.prefix {
+ if !rel_path.starts_with(prefix.as_str()) {
+ continue;
+ }
}
- if entry_type == EntryMode::FILE {
- meta.set_content_length(status.size);
-
- // Use LFS OID as ETag if available, otherwise use regular OID
- let etag = if let Some(lfs) = &status.lfs {
- &lfs.oid
- } else {
- &status.oid
- };
- meta.set_etag(etag);
- }
-
- let path = if entry_type == EntryMode::DIR {
- format!("{}/", &status.path)
- } else {
- status.path.clone()
- };
-
- ctx.entries.push_back(oio::Entry::new(
- &build_rel_path(&self.core.root, &path),
- meta,
- ));
+ ctx.entries.push_back(oio::Entry::new(&rel_path, meta));
}
Ok(())
}
}
-/// Parse the Link header to extract the next page URL.
-/// HuggingFace API returns pagination info in the Link header with rel="next".
-/// Example: <https://huggingface.co/api/models/.../tree?cursor=xxx>; rel="next"
-fn parse_link_header(headers: &http::HeaderMap) -> Option<String> {
- let link_header = headers.get(http::header::LINK)?;
- let link_str = link_header.to_str().ok()?;
-
- // Parse Link header format: <url>; rel="next"
- for link in link_str.split(',') {
- if link.contains("rel=\"next\"") || link.contains("rel='next'") {
- // Extract URL from <url> using split_once for cleaner parsing
- let (_, rest) = link.split_once('<')?;
- let (inside, _) = rest.split_once('>')?;
- return Some(inside.to_string());
- }
- }
-
- None
-}
-
#[cfg(test)]
mod tests {
+ use super::super::backend::test_utils::gpt2_operator;
use super::*;
- use http::HeaderMap;
- use http::HeaderValue;
#[test]
- fn test_parse_link_header_with_next() {
- let mut headers = HeaderMap::new();
- headers.insert(
- http::header::LINK,
- HeaderValue::from_static(
- r#"<https://huggingface.co/api/models/test/tree/main?cursor=abc123>; rel="next""#,
- ),
- );
-
- let result = parse_link_header(&headers);
- assert_eq!(
- result,
- Some("https://huggingface.co/api/models/test/tree/main?cursor=abc123".to_string())
- );
+ fn test_parse_next_cursor() {
+ let link =
+ r#"<https://huggingface.co/api/models/org/model/tree/main?cursor=abc123>; rel="next""#;
+ assert_eq!(parse_next_cursor(link), Some("abc123".to_string()));
}
#[test]
- fn test_parse_link_header_with_single_quotes() {
- let mut headers = HeaderMap::new();
- headers.insert(
- http::header::LINK,
- HeaderValue::from_static(
- r#"<https://huggingface.co/api/models/test/tree/main?cursor=xyz>; rel='next'"#,
- ),
- );
-
- let result = parse_link_header(&headers);
- assert_eq!(
- result,
- Some("https://huggingface.co/api/models/test/tree/main?cursor=xyz".to_string())
- );
+ fn test_parse_next_cursor_no_next() {
+ let link =
+ r#"<https://huggingface.co/api/models/org/model/tree/main?cursor=abc123>; rel="prev""#;
+ assert_eq!(parse_next_cursor(link), None);
}
- #[test]
- fn test_parse_link_header_without_next() {
- let mut headers = HeaderMap::new();
- headers.insert(
- http::header::LINK,
- HeaderValue::from_static(
- r#"<https://huggingface.co/api/models/test/tree/main>; rel="prev""#,
- ),
- );
-
- let result = parse_link_header(&headers);
- assert_eq!(result, None);
- }
-
- #[test]
- fn test_parse_link_header_multiple_links() {
- let mut headers = HeaderMap::new();
- headers.insert(
- http::header::LINK,
- HeaderValue::from_static(
- r#"<https://huggingface.co/api/prev>; rel="prev", <https://huggingface.co/api/next?cursor=456>; rel="next""#,
- ),
- );
-
- let result = parse_link_header(&headers);
- assert_eq!(
- result,
- Some("https://huggingface.co/api/next?cursor=456".to_string())
- );
- }
-
- #[test]
- fn test_parse_link_header_no_header() {
- let headers = HeaderMap::new();
- let result = parse_link_header(&headers);
- assert_eq!(result, None);
+ #[tokio::test]
+ async fn test_list_model_root() {
+ let op = gpt2_operator();
+ let entries = op.list("/").await.expect("list should succeed");
+ let names: Vec<&str> = entries.iter().map(|e| e.name()).collect();
+ assert!(names.contains(&"config.json"));
}
}
diff --git a/core/services/hf/src/reader.rs b/core/services/hf/src/reader.rs
new file mode 100644
index 000000000..ec74f26ee
--- /dev/null
+++ b/core/services/hf/src/reader.rs
@@ -0,0 +1,173 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use http::Response;
+use http::StatusCode;
+use http::header;
+
+use xet::xet_session::{SessionError, XetDownloadStream, XetFileInfo};
+
+use super::core::HfCore;
+use opendal_core::raw::*;
+use opendal_core::*;
+
+pub enum HfReader {
+ Http(HttpBody),
+ Xet(XetDownloadStream),
+}
+
+impl HfReader {
+ /// Create a reader, automatically choosing between XET and HTTP.
+ ///
+ /// Buckets always use XET. For other repo types, a HEAD request
+ /// probes for the `X-Xet-Hash` header. Files stored on XET are
+ /// downloaded via the CAS protocol; all others fall back to HTTP GET.
+ pub async fn try_new(core: &HfCore, path: &str, range: BytesRange) -> Result<Self> {
+ if let Some(xet_file) = core.maybe_xet_file(path).await? {
+ return Self::try_new_xet(core, &xet_file, range).await;
+ }
+
+ if core.repo.is_bucket() {
+ return Err(Error::new(
+ ErrorKind::Unexpected,
+ "bucket file is missing XET metadata",
+ ));
+ }
+
+ Self::try_new_http(core, path, range).await
+ }
+
+ pub async fn try_new_http(core: &HfCore, path: &str, range: BytesRange) -> Result<Self> {
+ let client = core.info.http_client();
+ let uri = core.uri(path);
+ let url = uri.resolve_url(&core.endpoint);
+
+ let mut req = core.request(http::Method::GET, &url, Operation::Read)?;
+
+ if !range.is_full() {
+ req = req.header(header::RANGE, range.to_header());
+ }
+
+ let req = req.body(Buffer::new()).map_err(new_request_build_error)?;
+
+ let resp = client.fetch(req).await?;
+ let status = resp.status();
+
+ match status {
+ StatusCode::OK | StatusCode::PARTIAL_CONTENT => Ok(Self::Http(resp.into_body())),
+ _ => {
+ let (part, mut body) = resp.into_parts();
+ let buf = body.to_buffer().await?;
+ Err(super::error::parse_error(Response::from_parts(part, buf)))
+ }
+ }
+ }
+
+ async fn try_new_xet(
+ core: &HfCore,
+ file_info: &XetFileInfo,
+ range: BytesRange,
+ ) -> Result<Self> {
+ let group = core.xet_download_group().await?;
+
+ let xet_range = if range.is_full() {
+ None
+ } else {
+ let start = range.offset();
+ let end = range.size().map(|s| start + s).unwrap_or(u64::MAX);
+ Some(start..end)
+ };
+
+ let mut stream = group
+ .download_stream(file_info.clone(), xet_range)
+ .await
+ .map_err(|err| {
+ Error::new(
+ ErrorKind::Unexpected,
+ "failed to create xet download stream",
+ )
+ .set_source(err)
+ })?;
+ stream.start();
+ Ok(Self::Xet(stream))
+ }
+}
+
+fn map_session_error(e: SessionError) -> Error {
+ Error::new(ErrorKind::Unexpected, "xet read error").set_source(e)
+}
+
+impl oio::Read for HfReader {
+ async fn read(&mut self) -> Result<Buffer> {
+ match self {
+ Self::Http(body) => body.read().await,
+ Self::Xet(stream) => match stream.next().await {
+ Ok(Some(bytes)) => Ok(Buffer::from(bytes)),
+ Ok(None) => Ok(Buffer::new()),
+ Err(e) => Err(map_session_error(e)),
+ },
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::super::backend::test_utils::{gpt2_operator, mbpp_operator};
+
+ /// Parquet magic bytes: "PAR1"
+ const PARQUET_MAGIC: &[u8] = b"PAR1";
+
+ #[tokio::test]
+ async fn test_read_model_config() {
+ let op = gpt2_operator();
+ let data = op.read("config.json").await.expect("read should succeed");
+ serde_json::from_slice::<serde_json::Value>(&data.to_vec())
+ .expect("config.json should be valid JSON");
+ }
+
+ /// Exercises the XET download code path against a public dataset known to
+ /// have XET-stored files. Behavior tests cannot reliably cover this path
+ /// because the test dataset may not contain any XET files.
+ #[tokio::test]
+ #[ignore = "requires network access"]
+ async fn test_read_xet_parquet() {
+ let op = mbpp_operator();
+ let data = op
+ .read("full/train-00000-of-00001.parquet")
+ .await
+ .expect("xet read should succeed");
+ let bytes = data.to_vec();
+ assert!(bytes.len() > 8);
+ assert_eq!(&bytes[..4], PARQUET_MAGIC);
+ assert_eq!(&bytes[bytes.len() - 4..], PARQUET_MAGIC);
+ }
+
+ /// Exercises XET range reads (XetDownloadStream with a byte range).
+ #[tokio::test]
+ #[ignore = "requires network access"]
+ async fn test_read_xet_range() {
+ let op = mbpp_operator();
+ let data = op
+ .read_with("full/train-00000-of-00001.parquet")
+ .range(0..4)
+ .await
+ .expect("xet range read should succeed");
+ let bytes = data.to_vec();
+ assert_eq!(bytes.len(), 4);
+ assert_eq!(&bytes, PARQUET_MAGIC);
+ }
+}
diff --git a/core/services/hf/src/uri.rs b/core/services/hf/src/uri.rs
new file mode 100644
index 000000000..452d2314e
--- /dev/null
+++ b/core/services/hf/src/uri.rs
@@ -0,0 +1,590 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use percent_encoding::{NON_ALPHANUMERIC, utf8_percent_encode};
+use serde::Deserialize;
+use serde::Serialize;
+
+use super::HUGGINGFACE_SCHEME;
+use opendal_core::raw::*;
+
+/// Repository type of Huggingface. Supports `model`, `dataset`, `space`, and `bucket`.
+/// [Reference](https://huggingface.co/docs/hub/repositories)
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
+#[serde(rename_all = "lowercase")]
+pub enum HfRepoType {
+ #[default]
+ Model,
+ Dataset,
+ Space,
+ Bucket,
+}
+
+impl HfRepoType {
+ pub fn parse(s: &str) -> opendal_core::Result<Self> {
+ match s.to_lowercase().replace(' ', "").as_str() {
+ "model" | "models" => Ok(Self::Model),
+ "dataset" | "datasets" => Ok(Self::Dataset),
+ "space" | "spaces" => Ok(Self::Space),
+ "bucket" | "buckets" => Ok(Self::Bucket),
+ other => Err(opendal_core::Error::new(
+ opendal_core::ErrorKind::ConfigInvalid,
+ format!("unknown repo type: {other}"),
+ )
+ .with_context("service", HUGGINGFACE_SCHEME)),
+ }
+ }
+
+ pub fn as_str(&self) -> &'static str {
+ match self {
+ Self::Model => "model",
+ Self::Dataset => "dataset",
+ Self::Space => "space",
+ Self::Bucket => "bucket",
+ }
+ }
+
+ pub fn as_plural_str(&self) -> &'static str {
+ match self {
+ Self::Model => "models",
+ Self::Dataset => "datasets",
+ Self::Space => "spaces",
+ Self::Bucket => "buckets",
+ }
+ }
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct HfRepo {
+ pub repo_type: HfRepoType,
+ pub repo_id: String,
+ pub revision: Option<String>,
+}
+
+impl HfRepo {
+ pub fn new(repo_type: HfRepoType, repo_id: String, revision: Option<String>) -> Self {
+ Self {
+ repo_type,
+ repo_id,
+ revision,
+ }
+ }
+
+ /// Whether this repo is a bucket (as opposed to a git-based repo).
+ pub fn is_bucket(&self) -> bool {
+ self.repo_type == HfRepoType::Bucket
+ }
+
+ /// Return the revision, defaulting to "main" if unset.
+ pub fn revision(&self) -> &str {
+ self.revision.as_deref().unwrap_or("main")
+ }
+
+ /// Create an `HfUri` for the given root and path within this repo.
+ pub fn uri(&self, root: &str, path: &str) -> HfUri {
+ HfUri {
+ repo: self.clone(),
+ path: build_abs_path(root, path)
+ .trim_start_matches('/')
+ .trim_end_matches('/')
+ .to_string(),
+ }
+ }
+
+ /// Build the paths-info API URL for this repository.
+ pub fn paths_info_url(&self, endpoint: &str) -> String {
+ match self.repo_type {
+ HfRepoType::Bucket => {
+ format!("{}/api/buckets/{}/paths-info", endpoint, &self.repo_id)
+ }
+ _ => {
+ format!(
+ "{}/api/{}/{}/paths-info/{}",
+ endpoint,
+ self.repo_type.as_plural_str(),
+ &self.repo_id,
+ percent_encode_revision(self.revision()),
+ )
+ }
+ }
+ }
+
+ /// Build the XET token API URL for this repository.
+ pub fn xet_token_url(&self, endpoint: &str, token_type: &str) -> String {
+ match self.repo_type {
+ HfRepoType::Bucket => {
+ format!(
+ "{}/api/buckets/{}/xet-{}-token",
+ endpoint, &self.repo_id, token_type
+ )
+ }
+ _ => {
+ format!(
+ "{}/api/{}/{}/xet-{}-token/{}",
+ endpoint,
+ self.repo_type.as_plural_str(),
+ &self.repo_id,
+ token_type,
+ self.revision(),
+ )
+ }
+ }
+ }
+
+ /// Build the bucket batch API URL for this repository.
+ pub fn bucket_batch_url(&self, endpoint: &str) -> String {
+ format!("{}/api/buckets/{}/batch", endpoint, &self.repo_id)
+ }
+
+ /// Build the git commit API URL for this repository.
+ pub fn git_commit_url(&self, endpoint: &str) -> String {
+ format!(
+ "{}/api/{}/{}/commit/{}",
+ endpoint,
+ self.repo_type.as_plural_str(),
+ &self.repo_id,
+ percent_encode_revision(self.revision()),
+ )
+ }
+}
+
+/// Parsed Hugging Face URI following the official format:
+/// `hf://[<repo_type_prefix>/]<repo_id>[@<revision>][/<path_in_repo>]`
+///
+/// Use this directly when you need access to `path_in_repo` separately
+/// from the config (e.g. to resolve a specific file within the repo).
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct HfUri {
+ pub repo: HfRepo,
+ pub path: String,
+}
+
+impl HfUri {
+ /// Parse a Hugging Face path into its components.
+ /// Path format: `[<repo_type>/]<repo_id>[@<revision>][/<path_in_repo>]`
+ pub fn parse(path: &str) -> opendal_core::Result<Self> {
+ if path.is_empty() {
+ return Err(opendal_core::Error::new(
+ opendal_core::ErrorKind::ConfigInvalid,
+ "repo_id is required in uri path",
+ )
+ .with_context("service", HUGGINGFACE_SCHEME));
+ }
+
+ let mut path = path.to_string();
+
+ // Strip repo_type prefix if present (e.g. "datasets/user/repo" → "user/repo")
+ let repo_type = if let Some((first, rest)) = path.split_once('/') {
+ if let Ok(rt) = HfRepoType::parse(first) {
+ path = rest.to_string();
+ rt
+ } else {
+ HfRepoType::Model
+ }
+ } else if HfRepoType::parse(&path).is_ok() {
+ return Err(opendal_core::Error::new(
+ opendal_core::ErrorKind::ConfigInvalid,
+ "repository name is required in uri path",
+ )
+ .with_context("service", HUGGINGFACE_SCHEME));
+ } else {
+ HfRepoType::Model
+ };
+
+ // Parse repo_id, revision, and path_in_repo.
+ // Path is now: <repo_id>[@<revision>][/<path_in_repo>]
+ let (repo_id, revision, path_in_repo) = if path.contains('/') {
+ // Check if @ appears in the first two segments (the repo_id portion).
+ // This distinguishes "user/repo@rev/file" from "user/repo/path/to/@file".
+ let first_two: String = path.splitn(3, '/').take(2).collect::<Vec<_>>().join("/");
+
+ if first_two.contains('@') {
+ let (repo_id, rev_and_path) = path.split_once('@').unwrap();
+ let rev_and_path = rev_and_path.replace("%2F", "/");
+ let (revision, path_in_repo) = Self::parse_revision(&rev_and_path);
+ (repo_id.to_string(), Some(revision), path_in_repo)
+ } else {
+ let segments: Vec<_> = path.splitn(3, '/').collect();
+ let repo_id = format!("{}/{}", segments[0], segments[1]);
+ let path_in_repo = segments.get(2).copied().unwrap_or("").to_string();
+ (repo_id, None, path_in_repo)
+ }
+ } else if let Some((repo_id, rev)) = path.split_once('@') {
+ let rev = rev.replace("%2F", "/");
+ (
+ repo_id.to_string(),
+ if rev.is_empty() { None } else { Some(rev) },
+ String::new(),
+ )
+ } else {
+ (path, None, String::new())
+ };
+
+ Ok(Self {
+ repo: HfRepo::new(repo_type, repo_id, revision),
+ path: path_in_repo,
+ })
+ }
+
+ /// Split a string after `@` into (revision, path_in_repo).
+ /// Handles special refs like `refs/convert/parquet` and `refs/pr/10`.
+ fn parse_revision(rev_and_path: &str) -> (String, String) {
+ if !rev_and_path.contains('/') {
+ return (rev_and_path.to_string(), String::new());
+ }
+
+ // Match special refs: refs/(convert|pr)/<segment>
+ if let Some(rest) = rev_and_path.strip_prefix("refs/convert/") {
+ return if let Some(slash) = rest.find('/') {
+ (
+ rev_and_path[..14 + slash].to_string(),
+ rest[slash + 1..].to_string(),
+ )
+ } else {
+ (rev_and_path.to_string(), String::new())
+ };
+ }
+ if let Some(rest) = rev_and_path.strip_prefix("refs/pr/") {
+ return if let Some(slash) = rest.find('/') {
+ let revision = format!("refs/pr/{}", &rest[..slash]);
+ (revision, rest[slash + 1..].to_string())
+ } else {
+ (rev_and_path.to_string(), String::new())
+ };
+ }
+
+ // Regular revision: split on first /
+ let (rev, path) = rev_and_path.split_once('/').unwrap();
+ (rev.to_string(), path.to_string())
+ }
+
+ /// Return the revision, defaulting to "main" if unset.
+ pub fn revision(&self) -> &str {
+ self.repo.revision()
+ }
+
+ /// Build the resolve URL for this URI.
+ pub fn resolve_url(&self, endpoint: &str) -> String {
+ let revision = percent_encode_revision(self.revision());
+ let path = percent_encode_path(&self.path);
+ match self.repo.repo_type {
+ HfRepoType::Model => {
+ format!(
+ "{}/{}/resolve/{}/{}",
+ endpoint, &self.repo.repo_id, revision, path
+ )
+ }
+ HfRepoType::Dataset => {
+ format!(
+ "{}/datasets/{}/resolve/{}/{}",
+ endpoint, &self.repo.repo_id, revision, path
+ )
+ }
+ HfRepoType::Space => {
+ format!(
+ "{}/spaces/{}/resolve/{}/{}",
+ endpoint, &self.repo.repo_id, revision, path
+ )
+ }
+ HfRepoType::Bucket => {
+ format!(
+ "{}/buckets/{}/resolve/{}",
+ endpoint, &self.repo.repo_id, path
+ )
+ }
+ }
+ }
+
+ /// Build the paths-info API URL for this URI.
+ pub fn paths_info_url(&self, endpoint: &str) -> String {
+ self.repo.paths_info_url(endpoint)
+ }
+
+ /// Build the file tree API URL for this URI.
+ pub fn file_tree_url(&self, endpoint: &str, recursive: bool, cursor: Option<&str>) -> String {
+ let mut url = if self.repo.is_bucket() {
+ format!(
+ "{}/api/buckets/{}/tree/{}?expand=True",
+ endpoint,
+ &self.repo.repo_id,
+ percent_encode_path(&self.path),
+ )
+ } else {
+ format!(
+ "{}/api/{}/{}/tree/{}/{}?expand=True",
+ endpoint,
+ self.repo.repo_type.as_plural_str(),
+ &self.repo.repo_id,
+ percent_encode_revision(self.revision()),
+ percent_encode_path(&self.path),
+ )
+ };
+
+ if recursive {
+ url.push_str("&recursive=True");
+ } else if self.repo.is_bucket() {
+ // Bucket tree API defaults to recursive; must opt out explicitly.
+ url.push_str("&recursive=false");
+ }
+
+ if let Some(cursor_val) = cursor {
+ url.push_str(&format!("&cursor={}", cursor_val));
+ }
+
+ url
+ }
+}
+
+pub(super) fn percent_encode_revision(revision: &str) -> String {
+ utf8_percent_encode(revision, NON_ALPHANUMERIC).to_string()
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn resolve(path: &str) -> HfUri {
+ HfUri::parse(path).unwrap()
+ }
+
+ #[test]
+ fn test_repo_type_parse() {
+ assert_eq!(HfRepoType::parse("models").unwrap(), HfRepoType::Model);
+ assert_eq!(HfRepoType::parse("Models").unwrap(), HfRepoType::Model);
+ assert_eq!(HfRepoType::parse("MODELS").unwrap(), HfRepoType::Model);
+ assert_eq!(HfRepoType::parse("datasets").unwrap(), HfRepoType::Dataset);
+ assert_eq!(HfRepoType::parse("Datasets").unwrap(), HfRepoType::Dataset);
+ assert_eq!(HfRepoType::parse("spaces").unwrap(), HfRepoType::Space);
+ assert_eq!(HfRepoType::parse("Spaces").unwrap(), HfRepoType::Space);
+ assert_eq!(HfRepoType::parse("model").unwrap(), HfRepoType::Model);
+ assert_eq!(HfRepoType::parse("dataset").unwrap(), HfRepoType::Dataset);
+ assert_eq!(HfRepoType::parse("space").unwrap(), HfRepoType::Space);
+ assert_eq!(HfRepoType::parse("data sets").unwrap(), HfRepoType::Dataset);
+ assert_eq!(HfRepoType::parse("Data Sets").unwrap(), HfRepoType::Dataset);
+ assert!(HfRepoType::parse("unknown").is_err());
+ assert!(HfRepoType::parse("foobar").is_err());
+ }
+
+ #[test]
+ fn resolve_with_namespace() {
+ let p = resolve("username/my_model");
+ assert_eq!(p.repo.repo_type, HfRepoType::Model);
+ assert_eq!(p.repo.repo_id, "username/my_model");
+ assert!(p.repo.revision.is_none());
+ assert_eq!(p.path, "");
+ }
+
+ #[test]
+ fn resolve_with_revision() {
+ let p = resolve("username/my_model@dev");
+ assert_eq!(p.repo.repo_type, HfRepoType::Model);
+ assert_eq!(p.repo.repo_id, "username/my_model");
+ assert_eq!(p.repo.revision.as_deref(), Some("dev"));
+ assert_eq!(p.path, "");
+ }
+
+ #[test]
+ fn resolve_datasets_prefix() {
+ let p = resolve("datasets/username/my_dataset");
+ assert_eq!(p.repo.repo_type, HfRepoType::Dataset);
+ assert_eq!(p.repo.repo_id, "username/my_dataset");
+ assert!(p.repo.revision.is_none());
+ assert_eq!(p.path, "");
+ }
+
+ #[test]
+ fn resolve_datasets_prefix_and_revision() {
+ let p = resolve("datasets/username/my_dataset@dev");
+ assert_eq!(p.repo.repo_type, HfRepoType::Dataset);
+ assert_eq!(p.repo.repo_id, "username/my_dataset");
+ assert_eq!(p.repo.revision.as_deref(), Some("dev"));
+ assert_eq!(p.path, "");
+ }
+
+ #[test]
+ fn resolve_with_path_in_repo() {
+ let p = resolve("username/my_model/config.json");
+ assert_eq!(p.repo.repo_type, HfRepoType::Model);
+ assert_eq!(p.repo.repo_id, "username/my_model");
+ assert!(p.repo.revision.is_none());
+ assert_eq!(p.path, "config.json");
+ }
+
+ #[test]
+ fn resolve_with_revision_and_path() {
+ let p = resolve("username/my_model@dev/path/to/file.txt");
+ assert_eq!(p.repo.repo_type, HfRepoType::Model);
+ assert_eq!(p.repo.repo_id, "username/my_model");
+ assert_eq!(p.repo.revision.as_deref(), Some("dev"));
+ assert_eq!(p.path, "path/to/file.txt");
+ }
+
+ #[test]
+ fn resolve_datasets_revision_and_path() {
+ let p = resolve("datasets/username/my_dataset@dev/train/data.csv");
+ assert_eq!(p.repo.repo_type, HfRepoType::Dataset);
+ assert_eq!(p.repo.repo_id, "username/my_dataset");
+ assert_eq!(p.repo.revision.as_deref(), Some("dev"));
+ assert_eq!(p.path, "train/data.csv");
+ }
+
+ #[test]
+ fn resolve_refs_convert_revision() {
+ let p = resolve("datasets/squad@refs/convert/parquet");
+ assert_eq!(p.repo.repo_type, HfRepoType::Dataset);
+ assert_eq!(p.repo.repo_id, "squad");
+ assert_eq!(p.repo.revision.as_deref(), Some("refs/convert/parquet"));
+ assert_eq!(p.path, "");
+ }
+
+ #[test]
+ fn resolve_refs_pr_revision() {
+ let p = resolve("username/my_model@refs/pr/10");
+ assert_eq!(p.repo.repo_type, HfRepoType::Model);
+ assert_eq!(p.repo.repo_id, "username/my_model");
+ assert_eq!(p.repo.revision.as_deref(), Some("refs/pr/10"));
+ assert_eq!(p.path, "");
+ }
+
+ #[test]
+ fn resolve_encoded_revision() {
+ let p = resolve("username/my_model@refs%2Fpr%2F10");
+ assert_eq!(p.repo.repo_type, HfRepoType::Model);
+ assert_eq!(p.repo.repo_id, "username/my_model");
+ assert_eq!(p.repo.revision.as_deref(), Some("refs/pr/10"));
+ assert_eq!(p.path, "");
+ }
+
+ #[test]
+ fn resolve_at_in_path_not_revision() {
+ let p = resolve("username/my_model/path/to/@not-a-revision.txt");
+ assert_eq!(p.repo.repo_type, HfRepoType::Model);
+ assert_eq!(p.repo.repo_id, "username/my_model");
+ assert!(p.repo.revision.is_none());
+ assert_eq!(p.path, "path/to/@not-a-revision.txt");
+ }
+
+ #[test]
+ fn resolve_bare_repo_type_fails() {
+ assert!(HfUri::parse("datasets").is_err());
+ assert!(HfUri::parse("").is_err());
+ }
+
+ #[test]
+ fn resolve_bare_repo_no_namespace() {
+ let p = resolve("gpt2");
+ assert_eq!(p.repo.repo_type, HfRepoType::Model);
+ assert_eq!(p.repo.repo_id, "gpt2");
+ assert!(p.repo.revision.is_none());
+ assert_eq!(p.path, "");
+ }
+
+ #[test]
+ fn resolve_bare_repo_with_revision() {
+ let p = resolve("gpt2@dev");
+ assert_eq!(p.repo.repo_type, HfRepoType::Model);
+ assert_eq!(p.repo.repo_id, "gpt2");
+ assert_eq!(p.repo.revision.as_deref(), Some("dev"));
+ assert_eq!(p.path, "");
+ }
+
+ #[test]
+ fn resolve_bare_dataset_no_namespace() {
+ let p = resolve("datasets/squad");
+ assert_eq!(p.repo.repo_type, HfRepoType::Dataset);
+ assert_eq!(p.repo.repo_id, "squad");
+ assert!(p.repo.revision.is_none());
+ assert_eq!(p.path, "");
+ }
+
+ #[test]
+ fn resolve_bare_dataset_with_revision() {
+ let p = resolve("datasets/squad@dev");
+ assert_eq!(p.repo.repo_type, HfRepoType::Dataset);
+ assert_eq!(p.repo.repo_id, "squad");
+ assert_eq!(p.repo.revision.as_deref(), Some("dev"));
+ assert_eq!(p.path, "");
+ }
+
+ #[test]
+ fn resolve_models_prefix() {
+ let p = resolve("models/username/my_model");
+ assert_eq!(p.repo.repo_type, HfRepoType::Model);
+ assert_eq!(p.repo.repo_id, "username/my_model");
+ assert!(p.repo.revision.is_none());
+ assert_eq!(p.path, "");
+ }
+
+ #[test]
+ fn resolve_spaces_prefix() {
+ let p = resolve("spaces/username/my_space");
+ assert_eq!(p.repo.repo_type, HfRepoType::Space);
+ assert_eq!(p.repo.repo_id, "username/my_space");
+ assert!(p.repo.revision.is_none());
+ assert_eq!(p.path, "");
+ }
+
+ #[test]
+ fn resolve_buckets_prefix() {
+ let p = resolve("buckets/username/my_bucket");
+ assert_eq!(p.repo.repo_type, HfRepoType::Bucket);
+ assert_eq!(p.repo.repo_id, "username/my_bucket");
+ assert!(p.repo.revision.is_none());
+ assert_eq!(p.path, "");
+ }
+
+ #[test]
+ fn resolve_buckets_with_path() {
+ let p = resolve("buckets/username/my_bucket/data/file.txt");
+ assert_eq!(p.repo.repo_type, HfRepoType::Bucket);
+ assert_eq!(p.repo.repo_id, "username/my_bucket");
+ assert!(p.repo.revision.is_none());
+ assert_eq!(p.path, "data/file.txt");
+ }
+
+ #[test]
+ fn test_bucket_resolve_url() {
+ let p = resolve("buckets/user/bucket/file.txt");
+ let url = p.resolve_url("https://huggingface.co");
+ assert_eq!(
+ url,
+ "https://huggingface.co/buckets/user/bucket/resolve/file.txt"
+ );
+ }
+
+ #[test]
+ fn test_bucket_xet_token_urls() {
+ let p = resolve("buckets/user/bucket");
+ let read_url = p.repo.xet_token_url("https://huggingface.co", "read");
+ let write_url = p.repo.xet_token_url("https://huggingface.co", "write");
+ assert_eq!(
+ read_url,
+ "https://huggingface.co/api/buckets/user/bucket/xet-read-token"
+ );
+ assert_eq!(
+ write_url,
+ "https://huggingface.co/api/buckets/user/bucket/xet-write-token"
+ );
+ }
+
+ #[test]
+ fn test_bucket_batch_url() {
+ let p = resolve("buckets/user/bucket");
+ let url = p.repo.bucket_batch_url("https://huggingface.co");
+ assert_eq!(url, "https://huggingface.co/api/buckets/user/bucket/batch");
+ }
+}
diff --git a/core/services/hf/src/writer.rs b/core/services/hf/src/writer.rs
new file mode 100644
index 000000000..601b72d5c
--- /dev/null
+++ b/core/services/hf/src/writer.rs
@@ -0,0 +1,153 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use super::core::{BucketOperation, HfCore, LfsFile};
+use opendal_core::raw::*;
+use opendal_core::*;
+use xet::error::XetError;
+use xet::xet_session::{Sha256Policy, XetFileInfo, XetStreamUpload, XetUploadCommit};
+
+/// Writer that always uses the XET protocol for uploads.
+pub struct HfWriter {
+ core: Arc<HfCore>,
+ path: String,
+ xet_commit: XetUploadCommit,
+ xet_stream: XetStreamUpload,
+}
+
+impl HfWriter {
+ /// Create a new writer for the given path.
+ ///
+ /// All uploads go through the XET protocol regardless of file size.
+ /// An XET upload commit and stream are created eagerly so that
+ /// subsequent [`write`](oio::Write::write) calls can stream data
+ /// directly into the XET CAS.
+ pub async fn try_new(core: Arc<HfCore>, path: String) -> Result<Self> {
+ let commit = core.xet_upload_commit().await?;
+ let stream = commit
+ .upload_stream(None, Sha256Policy::Compute)
+ .await
+ .map_err(|err| {
+ Error::new(ErrorKind::Unexpected, "failed to start xet upload stream")
+ .set_source(err)
+ })?;
+ Ok(HfWriter {
+ core,
+ path,
+ xet_commit: commit,
+ xet_stream: stream,
+ })
+ }
+
+ /// Finalize the XET upload and register the file with HF.
+ ///
+ /// 1. Commit data to XET CAS (may already be done by `stream.finish()`)
+ /// 2. Register the file via git LFS commit or bucket batch API
+ ///
+ /// Retries on transient CAS propagation delays.
+ async fn commit(&mut self, file_info: &XetFileInfo) -> Result<Metadata> {
+ // Finalize the XET CAS upload. May return AlreadyCompleted
+ // if stream.finish() already committed internally.
+ match self.xet_commit.commit().await {
+ Ok(_) | Err(XetError::AlreadyCompleted) => {}
+ Err(e) => {
+ return Err(
+ Error::new(ErrorKind::Unexpected, "failed to commit xet upload").set_source(e),
+ );
+ }
+ }
+
+ let content_length = file_info
+ .file_size()
+ .expect("file_size must be set after finish()");
+ let meta = Metadata::default().with_content_length(content_length);
+
+ let repo_path = self.core.repo_path(&self.path);
+ if self.core.repo.is_bucket() {
+ let xet_hash = file_info.hash().to_string();
+ self.core
+ .commit_bucket(vec![BucketOperation::AddFile {
+ path: repo_path,
+ xet_hash,
+ }])
+ .await?;
+ } else {
+ let sha256 = file_info.sha256().ok_or_else(|| {
+ Error::new(ErrorKind::Unexpected, "xet upload returned no sha256")
+ })?;
+ let lfs_file = LfsFile {
+ path: repo_path,
+ oid: sha256.to_string(),
+ algo: "sha256".to_string(),
+ size: content_length,
+ };
+ self.core
+ .commit_git(vec![], vec![lfs_file], vec![], vec![])
+ .await?;
+ }
+
+ Ok(meta)
+ }
+}
+
+impl oio::Write for HfWriter {
+ async fn write(&mut self, bs: Buffer) -> Result<()> {
+ self.xet_stream.write(bs.to_bytes()).await.map_err(|err| {
+ Error::new(ErrorKind::Unexpected, "failed to write xet chunk").set_source(err)
+ })
+ }
+
+ async fn close(&mut self) -> Result<Metadata> {
+ let file_meta = self.xet_stream.finish().await.map_err(|err| {
+ Error::new(ErrorKind::Unexpected, "failed to finish xet upload").set_source(err)
+ })?;
+
+ self.commit(&file_meta.xet_info).await
+ }
+
+ async fn abort(&mut self) -> Result<()> {
+ self.xet_stream.abort();
+ Ok(())
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::super::backend::test_utils::testing_bucket_operator;
+
+ /// Bucket writes always use XET and commit via NDJSON batch — this code
+ /// path is not covered by behavior tests unless run with a bucket config.
+ /// Requires HF_OPENDAL_BUCKET and HF_OPENDAL_TOKEN.
+ #[tokio::test]
+ #[ignore]
+ async fn test_bucket_write_roundtrip() {
+ let op = testing_bucket_operator();
+ let path = "tests/bucket-roundtrip.bin";
+ let content = b"Binary content for bucket roundtrip test";
+
+ op.write(path, content.as_slice())
+ .await
+ .expect("bucket write should succeed");
+
+ let data = op.read(path).await.expect("read should succeed");
+ assert_eq!(data.to_bytes().as_ref(), content);
+
+ let _ = op.delete(path).await;
+ }
+}
diff --git a/core/tests/behavior/async_delete.rs b/core/tests/behavior/async_delete.rs
index b1584badd..93384ac01 100644
--- a/core/tests/behavior/async_delete.rs
+++ b/core/tests/behavior/async_delete.rs
@@ -202,13 +202,28 @@ pub async fn test_remove_all_basic(op: Operator) -> Result<()> {
test_blocking_remove_all_with_objects(op, parent, ["a/b", "a/c", "a/d/e"]).await
}
-/// Remove all under a prefix, while the prefix itself is also an object
+/// Remove all under a prefix, while the prefix itself is also an object.
+///
+/// This test requires flat key storage where a path can be both a file
+/// and a directory prefix simultaneously (e.g., S3). Services with real
+/// directory semantics (e.g., git-based repos) cannot support this
+/// because a path cannot be both a file and a directory.
pub async fn test_remove_all_with_prefix_exists(op: Operator) -> Result<()> {
let parent = uuid::Uuid::new_v4().to_string();
let (content, _) = gen_bytes(op.info().full_capability());
op.write(&parent, content)
.await
.expect("write must succeed");
+
+ // Probe: write a file under the same path used as a prefix. This
+ // fails on services with real directory semantics.
+ let (content, _) = gen_bytes(op.info().full_capability());
+ if op.write(&format!("{parent}/probe"), content).await.is_err() {
+ let _ = op.delete(&parent).await;
+ return Ok(());
+ }
+ let _ = op.delete(&format!("{parent}/probe")).await;
+
test_blocking_remove_all_with_objects(op, parent, ["a", "a/b", "a/c", "a/b/e"]).await
}
diff --git a/core/tests/behavior/async_list.rs b/core/tests/behavior/async_list.rs
index fab890280..2a911ff74 100644
--- a/core/tests/behavior/async_list.rs
+++ b/core/tests/behavior/async_list.rs
@@ -112,6 +112,9 @@ pub async fn test_list_prefix(op: Operator) -> Result<()> {
/// listing a directory, which contains more objects than a single page can take.
pub async fn test_list_rich_dir(op: Operator) -> Result<()> {
+ if !op.info().full_capability().create_dir {
+ return Ok(());
+ }
// Gdrive think that this test is an abuse of their service and redirect us
// to an infinite loop. Let's ignore this test for gdrive.
#[cfg(feature = "services-gdrive")]
@@ -145,6 +148,9 @@ pub async fn test_list_rich_dir(op: Operator) -> Result<()> {
/// List empty dir should return itself.
pub async fn test_list_empty_dir(op: Operator) -> Result<()> {
+ if !op.info().full_capability().create_dir {
+ return Ok(());
+ }
let dir = format!("{}/", uuid::Uuid::new_v4());
op.create_dir(&dir).await.expect("write must succeed");
@@ -237,6 +243,9 @@ pub async fn test_list_non_exist_dir(op: Operator) -> Result<()> {
/// List dir should return correct sub dir.
pub async fn test_list_sub_dir(op: Operator) -> Result<()> {
+ if !op.info().full_capability().create_dir {
+ return Ok(());
+ }
let path = format!("{}/", uuid::Uuid::new_v4());
op.create_dir(&path).await.expect("create must succeed");
@@ -265,6 +274,9 @@ pub async fn test_list_sub_dir(op: Operator) -> Result<()> {
/// List dir should also to list nested dir.
pub async fn test_list_nested_dir(op: Operator) -> Result<()> {
+ if !op.info().full_capability().create_dir {
+ return Ok(());
+ }
let parent = format!("{}/", uuid::Uuid::new_v4());
op.create_dir(&parent)
.await
@@ -418,6 +430,10 @@ pub async fn test_list_non_exist_dir_with_recursive(op: Operator) -> Result<()>
}
pub async fn test_list_root_with_recursive(op: Operator) -> Result<()> {
+ if !op.info().full_capability().create_dir {
+ return Ok(());
+ }
+
op.create_dir("/").await?;
let w = op.lister_with("").recursive(true).await?;
@@ -435,6 +451,9 @@ pub async fn test_list_root_with_recursive(op: Operator) -> Result<()> {
// Walk top down should output as expected
pub async fn test_list_dir_with_recursive(op: Operator) -> Result<()> {
+ if !op.info().full_capability().create_dir {
+ return Ok(());
+ }
let parent = uuid::Uuid::new_v4().to_string();
let paths = [
@@ -473,6 +492,9 @@ pub async fn test_list_dir_with_recursive(op: Operator) -> Result<()> {
// same as test_list_dir_with_recursive except listing 'x' instead of 'x/'
pub async fn test_list_dir_with_recursive_no_trailing_slash(op: Operator) -> Result<()> {
+ if !op.info().full_capability().create_dir {
+ return Ok(());
+ }
let parent = uuid::Uuid::new_v4().to_string();
let paths = [
@@ -542,6 +564,9 @@ pub async fn test_list_file_with_recursive(op: Operator) -> Result<()> {
// Remove all should remove all in this path.
pub async fn test_remove_all(op: Operator) -> Result<()> {
+ if !op.info().full_capability().create_dir {
+ return Ok(());
+ }
let parent = uuid::Uuid::new_v4().to_string();
let expected = [