You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by xu...@apache.org on 2026/09/19 18:11:27 UTC
(opendal-yinyang) 01/01: feat(core): persist and read file contents
This is an automated email from the ASF dual-hosted git repository.
Xuanwo pushed a commit to branch xuanwo/file-content
in repository https://gitbox.apache.org/repos/asf/opendal-yinyang.git
commit eeb8caa272faec5d1890b1a77fffcbebaf8be279
Author: Xuanwo <gi...@xuanwo.io>
AuthorDate: Sun Sep 20 02:11:16 2026 +0800
feat(core): persist and read file contents
Co-authored-by: Chojan Shang <ps...@apache.org>
---
.github/workflows/ci.yml | 2 -
Cargo.lock | 1206 +++++++++++++++++++++++++++++++++++++++-
Cargo.toml | 1 +
crates/core/Cargo.toml | 9 +-
crates/core/src/content.rs | 197 +++++++
crates/core/src/error.rs | 5 +
crates/core/src/lib.rs | 1 +
crates/core/src/publication.rs | 4 +-
crates/core/tests/core.rs | 307 +++++++++-
specs/yinyang-format.md | 29 +
10 files changed, 1747 insertions(+), 14 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 3547ac7..27dd4a3 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -19,8 +19,6 @@ name: Apache OpenDAL™ YinYang CI
on:
pull_request:
- branches:
- - main
permissions:
contents: read
diff --git a/Cargo.lock b/Cargo.lock
index 21bd158..5b54d65 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -38,7 +38,7 @@ version = "1.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc"
dependencies = [
- "windows-sys",
+ "windows-sys 0.61.2",
]
[[package]]
@@ -49,7 +49,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d"
dependencies = [
"anstyle",
"once_cell_polyfill",
- "windows-sys",
+ "windows-sys 0.61.2",
]
[[package]]
@@ -70,10 +70,45 @@ version = "0.6.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94a214ba60d6231afd0e805e3c27c45a1626d9debaa5a5061c45a1ea1b2f1ed0"
dependencies = [
- "hashbrown",
+ "hashbrown 0.17.1",
"slab",
]
+[[package]]
+name = "atomic-waker"
+version = "1.1.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0"
+
+[[package]]
+name = "aws-lc-rs"
+version = "1.18.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b281d307588d634de920874890732659e2e7672f72b5e10e81badc1a8a83621e"
+dependencies = [
+ "aws-lc-sys",
+ "zeroize",
+]
+
+[[package]]
+name = "aws-lc-sys"
+version = "0.45.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9bff6c3b54fad79a2e60b8102caf565819711497c1f5f092f49508e2f5c31b27"
+dependencies = [
+ "cc",
+ "cmake",
+ "dunce",
+ "fs_extra",
+ "pkg-config",
+]
+
+[[package]]
+name = "base64"
+version = "0.22.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6"
+
[[package]]
name = "base64"
version = "0.23.1"
@@ -86,6 +121,12 @@ version = "1.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
+[[package]]
+name = "bitflags"
+version = "2.13.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06"
+
[[package]]
name = "blake3"
version = "1.8.7"
@@ -137,6 +178,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ad534f4357a5264cce5019c989cf66a4f0dc4e0d1b1d15f8aacec0ff7360273"
dependencies = [
"find-msvc-tools",
+ "jobserver",
+ "libc",
"shlex",
]
@@ -152,6 +195,17 @@ version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527"
+[[package]]
+name = "chacha20"
+version = "0.10.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06"
+dependencies = [
+ "cfg-if",
+ "cpufeatures",
+ "rand_core",
+]
+
[[package]]
name = "clap"
version = "4.6.6"
@@ -192,24 +246,85 @@ version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9"
+[[package]]
+name = "cmake"
+version = "0.1.58"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678"
+dependencies = [
+ "cc",
+]
+
+[[package]]
+name = "cmov"
+version = "0.5.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a"
+
[[package]]
name = "colorchoice"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
+[[package]]
+name = "combine"
+version = "4.6.8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cfc320937d09e6de266b31b9afb480f197d7a861be86be7cb2ea7e5d1bfffc5e"
+dependencies = [
+ "bytes",
+ "memchr",
+]
+
[[package]]
name = "const-oid"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c"
+[[package]]
+name = "const-random"
+version = "0.1.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359"
+dependencies = [
+ "const-random-macro",
+]
+
+[[package]]
+name = "const-random-macro"
+version = "0.1.16"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e"
+dependencies = [
+ "getrandom 0.2.17",
+ "once_cell",
+ "tiny-keccak",
+]
+
[[package]]
name = "constant_time_eq"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b"
+[[package]]
+name = "core-foundation"
+version = "0.10.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6"
+dependencies = [
+ "core-foundation-sys",
+ "libc",
+]
+
+[[package]]
+name = "core-foundation-sys"
+version = "0.8.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
+
[[package]]
name = "cpufeatures"
version = "0.3.1"
@@ -219,6 +334,32 @@ dependencies = [
"libc",
]
+[[package]]
+name = "crc-fast"
+version = "1.10.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e75b2483e97a5a7da73ac68a05b629f9c53cff58d8ed1c77866079e18b00dba5"
+dependencies = [
+ "digest 0.10.7",
+ "spin",
+]
+
+[[package]]
+name = "crunchy"
+version = "0.2.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5"
+
+[[package]]
+name = "crypto-common"
+version = "0.1.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
+dependencies = [
+ "generic-array",
+ "typenum",
+]
+
[[package]]
name = "crypto-common"
version = "0.2.2"
@@ -228,13 +369,22 @@ dependencies = [
"hybrid-array",
]
+[[package]]
+name = "ctutils"
+version = "0.4.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e"
+dependencies = [
+ "cmov",
+]
+
[[package]]
name = "defmt"
version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1"
dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
"defmt-macros",
]
@@ -259,6 +409,15 @@ dependencies = [
"thiserror",
]
+[[package]]
+name = "digest"
+version = "0.10.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
+dependencies = [
+ "crypto-common 0.1.7",
+]
+
[[package]]
name = "digest"
version = "0.11.3"
@@ -267,7 +426,8 @@ checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2"
dependencies = [
"block-buffer",
"const-oid",
- "crypto-common",
+ "crypto-common 0.2.2",
+ "ctutils",
]
[[package]]
@@ -281,6 +441,21 @@ dependencies = [
"syn 3.0.3",
]
+[[package]]
+name = "dlv-list"
+version = "0.5.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "442039f5147480ba31067cb00ada1adae6892028e40e45fc5de7b7df6dcc1b5f"
+dependencies = [
+ "const-random",
+]
+
+[[package]]
+name = "dunce"
+version = "1.0.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813"
+
[[package]]
name = "find-msvc-tools"
version = "0.1.11"
@@ -296,6 +471,12 @@ dependencies = [
"percent-encoding",
]
+[[package]]
+name = "fs_extra"
+version = "1.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
+
[[package]]
name = "futures"
version = "0.3.34"
@@ -304,6 +485,7 @@ checksum = "9a31d2a3fbaaeb2af2368bbdd904aa8e812d3c04a1ee10d3171f52d556e5d0a3"
dependencies = [
"futures-channel",
"futures-core",
+ "futures-executor",
"futures-io",
"futures-sink",
"futures-task",
@@ -326,6 +508,17 @@ version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e"
+[[package]]
+name = "futures-executor"
+version = "0.3.34"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "031b47cf1a3c6cc8bc2fc76cd437f521619387907d469316e7c0bc278f1f5432"
+dependencies = [
+ "futures-core",
+ "futures-task",
+ "futures-util",
+]
+
[[package]]
name = "futures-io"
version = "0.3.34"
@@ -372,6 +565,29 @@ dependencies = [
"slab",
]
+[[package]]
+name = "generic-array"
+version = "0.14.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
+dependencies = [
+ "typenum",
+ "version_check",
+]
+
+[[package]]
+name = "getrandom"
+version = "0.2.17"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
+dependencies = [
+ "cfg-if",
+ "js-sys",
+ "libc",
+ "wasi",
+ "wasm-bindgen",
+]
+
[[package]]
name = "getrandom"
version = "0.4.3"
@@ -379,10 +595,19 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099"
dependencies = [
"cfg-if",
+ "js-sys",
"libc",
"r-efi",
+ "rand_core",
+ "wasm-bindgen",
]
+[[package]]
+name = "hashbrown"
+version = "0.14.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1"
+
[[package]]
name = "hashbrown"
version = "0.17.1"
@@ -395,6 +620,21 @@ version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
+[[package]]
+name = "hex"
+version = "0.4.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
+
+[[package]]
+name = "hmac"
+version = "0.13.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6303bc9732ae41b04cb554b844a762b4115a61bfaa81e3e83050991eeb56863f"
+dependencies = [
+ "digest 0.11.3",
+]
+
[[package]]
name = "http"
version = "1.5.0"
@@ -405,6 +645,35 @@ dependencies = [
"itoa",
]
+[[package]]
+name = "http-body"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ca2a8f2913ee65f60facd6a5905613afaa448497a0230cc41ce022d93290bc2c"
+dependencies = [
+ "bytes",
+ "http",
+]
+
+[[package]]
+name = "http-body-util"
+version = "0.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "23169fe34a5fbcdd3f3862e78fb9b6fccd5f02a6dc6f732547005d45631ce71c"
+dependencies = [
+ "bytes",
+ "futures-core",
+ "http",
+ "http-body",
+ "pin-project-lite",
+]
+
+[[package]]
+name = "httparse"
+version = "1.10.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87"
+
[[package]]
name = "hybrid-array"
version = "0.4.14"
@@ -414,6 +683,64 @@ dependencies = [
"typenum",
]
+[[package]]
+name = "hyper"
+version = "1.11.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "27b501faa50e7a26c3d3560ca625132f4078a17771f4810baf70475ae48cbe43"
+dependencies = [
+ "atomic-waker",
+ "bytes",
+ "futures-channel",
+ "futures-core",
+ "http",
+ "http-body",
+ "httparse",
+ "itoa",
+ "pin-project-lite",
+ "smallvec",
+ "tokio",
+ "want",
+]
+
+[[package]]
+name = "hyper-rustls"
+version = "0.27.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f"
+dependencies = [
+ "http",
+ "hyper",
+ "hyper-util",
+ "rustls",
+ "tokio",
+ "tokio-rustls",
+ "tower-service",
+]
+
+[[package]]
+name = "hyper-util"
+version = "0.1.20"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0"
+dependencies = [
+ "base64 0.22.1",
+ "bytes",
+ "futures-channel",
+ "futures-util",
+ "http",
+ "http-body",
+ "hyper",
+ "ipnet",
+ "libc",
+ "percent-encoding",
+ "pin-project-lite",
+ "socket2",
+ "tokio",
+ "tower-service",
+ "tracing",
+]
+
[[package]]
name = "icu_collections"
version = "2.3.0"
@@ -518,6 +845,12 @@ dependencies = [
"icu_properties",
]
+[[package]]
+name = "ipnet"
+version = "2.12.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
+
[[package]]
name = "is_terminal_polyfill"
version = "1.70.2"
@@ -585,6 +918,65 @@ dependencies = [
"jiff-tzdb",
]
+[[package]]
+name = "jni"
+version = "0.22.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498"
+dependencies = [
+ "cfg-if",
+ "combine",
+ "jni-macros",
+ "jni-sys",
+ "log",
+ "simd_cesu8",
+ "thiserror",
+ "walkdir",
+ "windows-link",
+]
+
+[[package]]
+name = "jni-macros"
+version = "0.22.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "rustc_version",
+ "simd_cesu8",
+ "syn 2.0.119",
+]
+
+[[package]]
+name = "jni-sys"
+version = "0.4.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2"
+dependencies = [
+ "jni-sys-macros",
+]
+
+[[package]]
+name = "jni-sys-macros"
+version = "0.4.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264"
+dependencies = [
+ "quote",
+ "syn 2.0.119",
+]
+
+[[package]]
+name = "jobserver"
+version = "0.1.35"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3"
+dependencies = [
+ "getrandom 0.4.3",
+ "libc",
+]
+
[[package]]
name = "js-sys"
version = "0.3.104"
@@ -614,6 +1006,12 @@ version = "0.4.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6"
+[[package]]
+name = "lru-slab"
+version = "0.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4050469837a6ff301cd14c1f8f24f88549e6d548f24f64e2148eb0f72cebc51f"
+
[[package]]
name = "md-5"
version = "0.11.0"
@@ -621,7 +1019,17 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98"
dependencies = [
"cfg-if",
- "digest",
+ "digest 0.11.3",
+]
+
+[[package]]
+name = "mea"
+version = "0.6.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c709842c4ce65cb91e2666ad5319dfc1efc3af0d34f02075eddca9000d9f8afb"
+dependencies = [
+ "hashbrown 0.17.1",
+ "slab",
]
[[package]]
@@ -630,6 +1038,17 @@ version = "2.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
+[[package]]
+name = "mio"
+version = "1.2.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4b18443e9c262bfe8fa82f51666e2642c53393f7e5c27b3e1aeab922cff5b9d8"
+dependencies = [
+ "libc",
+ "wasi",
+ "windows-sys 0.61.2",
+]
+
[[package]]
name = "once_cell"
version = "1.21.4"
@@ -649,6 +1068,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "33dbff14cc9bb085224256d6a81289d2f3202e85b06f408d42534b42162a4231"
dependencies = [
"opendal-core",
+ "opendal-http-transport-reqwest",
+ "opendal-service-s3",
]
[[package]]
@@ -659,7 +1080,7 @@ checksum = "48dbcef97d3eb7591db2c18d5cae95c836bcce07359b98d98dd6f4e861eb77b7"
dependencies = [
"anyhow",
"asyncband",
- "base64",
+ "base64 0.23.1",
"bytes",
"futures",
"http",
@@ -668,6 +1089,7 @@ dependencies = [
"md-5",
"percent-encoding",
"quick-xml",
+ "reqsign-core",
"serde",
"serde_json",
"tokio",
@@ -676,6 +1098,57 @@ dependencies = [
"web-time",
]
+[[package]]
+name = "opendal-http-transport-reqwest"
+version = "0.58.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "85663452ea32bbc17e8f79ab29788c846d116ec7de31451be9c787e462dcb36c"
+dependencies = [
+ "bytes",
+ "futures",
+ "http",
+ "http-body",
+ "opendal-core",
+ "reqwest",
+]
+
+[[package]]
+name = "opendal-service-s3"
+version = "0.58.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c64335f9f24ccb62ac36f1d976342b48611a75ba61979813a4f78a4ebd94de42"
+dependencies = [
+ "base64 0.23.1",
+ "bytes",
+ "crc-fast",
+ "http",
+ "log",
+ "md-5",
+ "opendal-core",
+ "quick-xml",
+ "reqsign-aws-v4",
+ "reqsign-core",
+ "reqsign-file-read-tokio",
+ "serde",
+ "url",
+]
+
+[[package]]
+name = "openssl-probe"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"
+
+[[package]]
+name = "ordered-multimap"
+version = "0.7.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "49203cdcae0030493bad186b28da2fa25645fa276a51b6fec8010d281e02ef79"
+dependencies = [
+ "dlv-list",
+ "hashbrown 0.14.5",
+]
+
[[package]]
name = "percent-encoding"
version = "2.3.2"
@@ -688,6 +1161,12 @@ version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"
+[[package]]
+name = "pkg-config"
+version = "0.3.34"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"
+
[[package]]
name = "portable-atomic"
version = "1.15.0"
@@ -731,6 +1210,63 @@ dependencies = [
"serde",
]
+[[package]]
+name = "quinn"
+version = "0.11.12"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4051e23e9185c255a7e33ef59cdbca87a22d359052eecd22fc6b901fb37d9d11"
+dependencies = [
+ "bytes",
+ "cfg_aliases",
+ "pin-project-lite",
+ "quinn-proto",
+ "quinn-udp",
+ "rustc-hash",
+ "rustls",
+ "socket2",
+ "thiserror",
+ "tokio",
+ "tracing",
+ "web-time",
+]
+
+[[package]]
+name = "quinn-proto"
+version = "0.11.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a9746dbde176634f4f2f1faf2404e30a31b2bc1e9cafb5329c95d8177a18c9fc"
+dependencies = [
+ "aws-lc-rs",
+ "bytes",
+ "getrandom 0.4.3",
+ "lru-slab",
+ "rand",
+ "rand_pcg",
+ "ring",
+ "rustc-hash",
+ "rustls",
+ "rustls-pki-types",
+ "slab",
+ "thiserror",
+ "tinyvec",
+ "tracing",
+ "web-time",
+]
+
+[[package]]
+name = "quinn-udp"
+version = "0.5.15"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694"
+dependencies = [
+ "cfg_aliases",
+ "libc",
+ "once_cell",
+ "socket2",
+ "tracing",
+ "windows-sys 0.61.2",
+]
+
[[package]]
name = "quote"
version = "1.0.47"
@@ -746,12 +1282,312 @@ version = "6.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
+[[package]]
+name = "rand"
+version = "0.10.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80"
+dependencies = [
+ "chacha20",
+ "getrandom 0.4.3",
+ "rand_core",
+]
+
+[[package]]
+name = "rand_core"
+version = "0.10.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69"
+
+[[package]]
+name = "rand_pcg"
+version = "0.10.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a"
+dependencies = [
+ "rand_core",
+]
+
+[[package]]
+name = "reqsign-aws-core"
+version = "3.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bac4749b7dfa7bfaccd01eb03e9dc795ed37e3f20d6f0f38e2c67ee85ad6bc86"
+dependencies = [
+ "bytes",
+ "form_urlencoded",
+ "hex",
+ "http",
+ "log",
+ "percent-encoding",
+ "quick-xml",
+ "reqsign-core",
+ "rust-ini",
+ "serde",
+ "serde_json",
+ "serde_urlencoded",
+ "sha1",
+]
+
+[[package]]
+name = "reqsign-aws-v4"
+version = "3.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ff250f0fd0b913fbd565e405acc553da0f13bde30bfb5403178c9d0313cdc15f"
+dependencies = [
+ "bytes",
+ "http",
+ "log",
+ "quick-xml",
+ "reqsign-aws-core",
+ "reqsign-core",
+ "serde",
+]
+
+[[package]]
+name = "reqsign-core"
+version = "3.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ff052daffb0599681c50f85c59e7236438976efe991ab864edd9f3b235501a0f"
+dependencies = [
+ "anyhow",
+ "base64 0.23.1",
+ "bytes",
+ "futures",
+ "hex",
+ "hmac",
+ "http",
+ "jiff",
+ "log",
+ "mea",
+ "percent-encoding",
+ "sha1",
+ "sha2",
+ "windows-sys 0.61.2",
+]
+
+[[package]]
+name = "reqsign-file-read-tokio"
+version = "3.0.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b3235df90a6bca681aa47dd86f2393d122a6d77042aa8a7c81e218cd45c5bfc0"
+dependencies = [
+ "anyhow",
+ "reqsign-core",
+ "tokio",
+]
+
+[[package]]
+name = "reqwest"
+version = "0.13.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "16a1cfa75cc186dd73d5818e510e042e40927bccc9c236b061cea97e1eb08029"
+dependencies = [
+ "base64 0.23.1",
+ "bytes",
+ "futures-core",
+ "futures-util",
+ "http",
+ "http-body",
+ "http-body-util",
+ "hyper",
+ "hyper-rustls",
+ "hyper-util",
+ "js-sys",
+ "log",
+ "percent-encoding",
+ "pin-project-lite",
+ "quinn",
+ "rustls",
+ "rustls-pki-types",
+ "rustls-platform-verifier",
+ "sync_wrapper",
+ "tokio",
+ "tokio-rustls",
+ "tokio-util",
+ "tower",
+ "tower-http",
+ "tower-service",
+ "url",
+ "wasm-bindgen",
+ "wasm-bindgen-futures",
+ "wasm-streams",
+ "web-sys",
+]
+
+[[package]]
+name = "ring"
+version = "0.17.14"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
+dependencies = [
+ "cc",
+ "cfg-if",
+ "getrandom 0.2.17",
+ "libc",
+ "untrusted",
+ "windows-sys 0.52.0",
+]
+
+[[package]]
+name = "rust-ini"
+version = "0.21.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "796e8d2b6696392a43bea58116b667fb4c29727dc5abd27d6acf338bb4f688c7"
+dependencies = [
+ "cfg-if",
+ "ordered-multimap",
+]
+
+[[package]]
+name = "rustc-hash"
+version = "2.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d"
+
+[[package]]
+name = "rustc_version"
+version = "0.4.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92"
+dependencies = [
+ "semver",
+]
+
+[[package]]
+name = "rustls"
+version = "0.23.45"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634"
+dependencies = [
+ "aws-lc-rs",
+ "once_cell",
+ "rustls-pki-types",
+ "rustls-webpki",
+ "subtle",
+ "zeroize",
+]
+
+[[package]]
+name = "rustls-native-certs"
+version = "0.8.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "dab5152771c58876a2146916e53e35057e1a4dfa2b9df0f0305b07f611fdea4d"
+dependencies = [
+ "openssl-probe",
+ "rustls-pki-types",
+ "schannel",
+ "security-framework",
+]
+
+[[package]]
+name = "rustls-pki-types"
+version = "1.15.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96"
+dependencies = [
+ "web-time",
+ "zeroize",
+]
+
+[[package]]
+name = "rustls-platform-verifier"
+version = "0.7.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0"
+dependencies = [
+ "core-foundation",
+ "core-foundation-sys",
+ "jni",
+ "log",
+ "once_cell",
+ "rustls",
+ "rustls-native-certs",
+ "rustls-platform-verifier-android",
+ "rustls-webpki",
+ "security-framework",
+ "security-framework-sys",
+ "webpki-root-certs",
+ "windows-sys 0.61.2",
+]
+
+[[package]]
+name = "rustls-platform-verifier-android"
+version = "0.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f"
+
+[[package]]
+name = "rustls-webpki"
+version = "0.103.15"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f3c3cf1d8b1e7d4927e2d154c3fcb02979afb9939629c62cd9048d4f07b60ac2"
+dependencies = [
+ "aws-lc-rs",
+ "ring",
+ "rustls-pki-types",
+ "untrusted",
+]
+
[[package]]
name = "rustversion"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
+[[package]]
+name = "ryu"
+version = "1.0.23"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
+
+[[package]]
+name = "same-file"
+version = "1.0.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502"
+dependencies = [
+ "winapi-util",
+]
+
+[[package]]
+name = "schannel"
+version = "0.1.29"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939"
+dependencies = [
+ "windows-sys 0.61.2",
+]
+
+[[package]]
+name = "security-framework"
+version = "3.7.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d"
+dependencies = [
+ "bitflags 2.13.2",
+ "core-foundation",
+ "core-foundation-sys",
+ "libc",
+ "security-framework-sys",
+]
+
+[[package]]
+name = "security-framework-sys"
+version = "2.17.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3"
+dependencies = [
+ "core-foundation-sys",
+ "libc",
+]
+
+[[package]]
+name = "semver"
+version = "1.0.28"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd"
+
[[package]]
name = "serde"
version = "1.0.229"
@@ -795,12 +1631,62 @@ dependencies = [
"zmij",
]
+[[package]]
+name = "serde_urlencoded"
+version = "0.7.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd"
+dependencies = [
+ "form_urlencoded",
+ "itoa",
+ "ryu",
+ "serde",
+]
+
+[[package]]
+name = "sha1"
+version = "0.11.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214"
+dependencies = [
+ "cfg-if",
+ "cpufeatures",
+ "digest 0.11.3",
+]
+
+[[package]]
+name = "sha2"
+version = "0.11.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4"
+dependencies = [
+ "cfg-if",
+ "cpufeatures",
+ "digest 0.11.3",
+]
+
[[package]]
name = "shlex"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
+[[package]]
+name = "simd_cesu8"
+version = "1.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "11031e251abf8611c80f460e19dbdeb54a66db918e49c65a7065b46ac7aec520"
+dependencies = [
+ "rustc_version",
+ "simdutf8",
+]
+
+[[package]]
+name = "simdutf8"
+version = "0.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
+
[[package]]
name = "slab"
version = "0.4.12"
@@ -813,6 +1699,22 @@ version = "1.16.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9be42f50aa861c555654aa3a37f52f4b1074bacf4e48fe0ef7fa584e80f1f0f"
+[[package]]
+name = "socket2"
+version = "0.6.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4"
+dependencies = [
+ "libc",
+ "windows-sys 0.61.2",
+]
+
+[[package]]
+name = "spin"
+version = "0.10.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "023a211cb3138dbc438680b32560ad89f699977624c9f8dbb95a47d5b4c07dd3"
+
[[package]]
name = "stable_deref_trait"
version = "1.2.1"
@@ -825,6 +1727,12 @@ version = "0.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
+[[package]]
+name = "subtle"
+version = "2.6.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
+
[[package]]
name = "syn"
version = "2.0.119"
@@ -847,6 +1755,15 @@ dependencies = [
"unicode-ident",
]
+[[package]]
+name = "sync_wrapper"
+version = "1.0.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263"
+dependencies = [
+ "futures-core",
+]
+
[[package]]
name = "synstructure"
version = "0.13.2"
@@ -878,6 +1795,15 @@ dependencies = [
"syn 3.0.3",
]
+[[package]]
+name = "tiny-keccak"
+version = "2.0.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237"
+dependencies = [
+ "crunchy",
+]
+
[[package]]
name = "tinystr"
version = "0.8.4"
@@ -910,8 +1836,12 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed"
dependencies = [
"bytes",
+ "libc",
+ "mio",
"pin-project-lite",
+ "socket2",
"tokio-macros",
+ "windows-sys 0.61.2",
]
[[package]]
@@ -925,6 +1855,99 @@ dependencies = [
"syn 3.0.3",
]
+[[package]]
+name = "tokio-rustls"
+version = "0.26.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b0c85f2c3ef0b1cd58b36682f4b17aaa995f0e5db534d85692b4903abce21f67"
+dependencies = [
+ "rustls",
+ "tokio",
+]
+
+[[package]]
+name = "tokio-util"
+version = "0.7.19"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "494815d09bf52b5548659851081238f0ca39ff638363907596da739561c62c52"
+dependencies = [
+ "bytes",
+ "futures-core",
+ "futures-sink",
+ "pin-project-lite",
+ "tokio",
+]
+
+[[package]]
+name = "tower"
+version = "0.5.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4"
+dependencies = [
+ "futures-core",
+ "futures-util",
+ "pin-project-lite",
+ "sync_wrapper",
+ "tokio",
+ "tower-layer",
+ "tower-service",
+]
+
+[[package]]
+name = "tower-http"
+version = "0.6.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840"
+dependencies = [
+ "bitflags 2.13.2",
+ "bytes",
+ "futures-util",
+ "http",
+ "http-body",
+ "pin-project-lite",
+ "tower",
+ "tower-layer",
+ "tower-service",
+ "url",
+]
+
+[[package]]
+name = "tower-layer"
+version = "0.3.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e"
+
+[[package]]
+name = "tower-service"
+version = "0.3.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3"
+
+[[package]]
+name = "tracing"
+version = "0.1.44"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100"
+dependencies = [
+ "pin-project-lite",
+ "tracing-core",
+]
+
+[[package]]
+name = "tracing-core"
+version = "0.1.36"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a"
+dependencies = [
+ "once_cell",
+]
+
+[[package]]
+name = "try-lock"
+version = "0.2.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
+
[[package]]
name = "typenum"
version = "1.20.1"
@@ -952,6 +1975,12 @@ dependencies = [
"tinyvec",
]
+[[package]]
+name = "untrusted"
+version = "0.9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
+
[[package]]
name = "url"
version = "2.5.8"
@@ -982,12 +2011,43 @@ version = "1.26.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b5772d71c9be8a8a6ac2117d949c5b224c1b72241bb611d9a3012edcf8af7812"
dependencies = [
- "getrandom",
+ "getrandom 0.4.3",
"js-sys",
"serde_core",
"wasm-bindgen",
]
+[[package]]
+name = "version_check"
+version = "0.9.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
+
+[[package]]
+name = "walkdir"
+version = "2.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b"
+dependencies = [
+ "same-file",
+ "winapi-util",
+]
+
+[[package]]
+name = "want"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e"
+dependencies = [
+ "try-lock",
+]
+
+[[package]]
+name = "wasi"
+version = "0.11.1+wasi-snapshot-preview1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
+
[[package]]
name = "wasm-bindgen"
version = "0.2.127"
@@ -1001,6 +2061,16 @@ dependencies = [
"wasm-bindgen-shared",
]
+[[package]]
+name = "wasm-bindgen-futures"
+version = "0.4.77"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6b7777d5cc23d0e91404e53ce2d5e8ec7acae3026b16233dba62cd3246457950"
+dependencies = [
+ "js-sys",
+ "wasm-bindgen",
+]
+
[[package]]
name = "wasm-bindgen-macro"
version = "0.2.127"
@@ -1033,6 +2103,29 @@ dependencies = [
"unicode-ident",
]
+[[package]]
+name = "wasm-streams"
+version = "0.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9d1ec4f6517c9e11ae630e200b2b65d193279042e28edd4a2cda233e46670bbb"
+dependencies = [
+ "futures-util",
+ "js-sys",
+ "wasm-bindgen",
+ "wasm-bindgen-futures",
+ "web-sys",
+]
+
+[[package]]
+name = "web-sys"
+version = "0.3.104"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c435338968042f4f59a557f690a253676d47ce13ceb55d70100e7facf6620a30"
+dependencies = [
+ "js-sys",
+ "wasm-bindgen",
+]
+
[[package]]
name = "web-time"
version = "1.1.0"
@@ -1043,6 +2136,15 @@ dependencies = [
"wasm-bindgen",
]
+[[package]]
+name = "webpki-root-certs"
+version = "1.0.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b96554aa2acc8ccdb7e1c9a58a7a68dd5d13bccc69cd124cb09406db612a1c9b"
+dependencies = [
+ "rustls-pki-types",
+]
+
[[package]]
name = "which"
version = "8.0.5"
@@ -1052,12 +2154,30 @@ dependencies = [
"libc",
]
+[[package]]
+name = "winapi-util"
+version = "0.1.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
+dependencies = [
+ "windows-sys 0.61.2",
+]
+
[[package]]
name = "windows-link"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
+[[package]]
+name = "windows-sys"
+version = "0.52.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
+dependencies = [
+ "windows-targets",
+]
+
[[package]]
name = "windows-sys"
version = "0.61.2"
@@ -1067,6 +2187,70 @@ dependencies = [
"windows-link",
]
+[[package]]
+name = "windows-targets"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
+dependencies = [
+ "windows_aarch64_gnullvm",
+ "windows_aarch64_msvc",
+ "windows_i686_gnu",
+ "windows_i686_gnullvm",
+ "windows_i686_msvc",
+ "windows_x86_64_gnu",
+ "windows_x86_64_gnullvm",
+ "windows_x86_64_msvc",
+]
+
+[[package]]
+name = "windows_aarch64_gnullvm"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
+
+[[package]]
+name = "windows_aarch64_msvc"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
+
+[[package]]
+name = "windows_i686_gnu"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
+
+[[package]]
+name = "windows_i686_gnullvm"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
+
+[[package]]
+name = "windows_i686_msvc"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
+
+[[package]]
+name = "windows_x86_64_gnu"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
+
+[[package]]
+name = "windows_x86_64_gnullvm"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
+
+[[package]]
+name = "windows_x86_64_msvc"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
+
[[package]]
name = "writeable"
version = "0.6.4"
@@ -1143,6 +2327,12 @@ dependencies = [
"synstructure",
]
+[[package]]
+name = "zeroize"
+version = "1.9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e"
+
[[package]]
name = "zerotrie"
version = "0.2.5"
diff --git a/Cargo.toml b/Cargo.toml
index ce7227f..e209a37 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -47,6 +47,7 @@ borsh = { version = "1.8.1", default-features = false, features = ["std"] }
clap = { version = "4.6.5", features = ["derive"] }
futures-util = "0.3"
opendal = { version = "0.58.2", default-features = false }
+tokio = { version = "1", features = ["io-util"] }
unicode-casefold = "0.2"
unicode-normalization = "0.1"
uuid = { version = "1", features = ["v4"] }
diff --git a/crates/core/Cargo.toml b/crates/core/Cargo.toml
index 9b54dac..497f6e1 100644
--- a/crates/core/Cargo.toml
+++ b/crates/core/Cargo.toml
@@ -30,13 +30,18 @@ blake3.workspace = true
borsh.workspace = true
futures-util.workspace = true
opendal.workspace = true
+tokio.workspace = true
unicode-casefold.workspace = true
unicode-normalization.workspace = true
uuid.workspace = true
[dev-dependencies]
-opendal = { workspace = true, features = ["services-memory"] }
-tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
+opendal = { workspace = true, features = [
+ "http-transport-reqwest",
+ "services-memory",
+ "services-s3",
+] }
+tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
[lints]
workspace = true
diff --git a/crates/core/src/content.rs b/crates/core/src/content.rs
new file mode 100644
index 0000000..a2fc81f
--- /dev/null
+++ b/crates/core/src/content.rs
@@ -0,0 +1,197 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::BTreeMap;
+
+use futures_util::TryStreamExt as _;
+use tokio::io::{AsyncRead, AsyncReadExt as _, AsyncWrite, AsyncWriteExt as _};
+
+use crate::{BlobRef, ContentId, Error, File, FilePart, Fs, NodeBody, Result, Tree};
+
+const DATA_PREFIX: &str = ".yinyang/data/";
+const BUFFER_BYTES: usize = 256 * 1024;
+
+impl Fs {
+ /// Upload a complete file without publishing a namespace change.
+ ///
+ /// Returns only after the backend acknowledges the immutable object.
+ /// Failed or cancelled uploads can leave unreachable data. Empty files do
+ /// not allocate an object.
+ pub async fn write_file(&self, source: &mut (impl AsyncRead + Unpin)) -> Result<File> {
+ let mut bytes = vec![0; BUFFER_BYTES];
+ let first = source
+ .read(&mut bytes)
+ .await
+ .map_err(|error| Error::from_io("read YinYang file source", error))?;
+ if first == 0 {
+ return File::new(ContentId::new(blake3::hash(&[]).into(), 0), Vec::new());
+ }
+ let path = format!("{DATA_PREFIX}{}", uuid::Uuid::new_v4().simple());
+ let mut writer = self
+ .operator
+ .writer_with(&path)
+ .if_not_exists(true)
+ .await
+ .map_err(|error| Error::from_storage("write YinYang file", error))?;
+ let result = async {
+ let mut hasher = blake3::Hasher::new();
+ let mut length = 0_u64;
+ let mut count = first;
+ loop {
+ length = length
+ .checked_add(count as u64)
+ .ok_or_else(|| Error::invalid("write YinYang file", "file length overflows"))?;
+ hasher.update(&bytes[..count]);
+ writer
+ .write(bytes[..count].to_vec())
+ .await
+ .map_err(|error| Error::from_storage("write YinYang file", error))?;
+ count = source
+ .read(&mut bytes)
+ .await
+ .map_err(|error| Error::from_io("read YinYang file source", error))?;
+ if count == 0 {
+ break;
+ }
+ }
+ writer
+ .close()
+ .await
+ .map_err(|error| Error::from_storage("close YinYang file", error))?;
+ let content = ContentId::new(hasher.finalize().into(), length);
+ File::new(
+ content,
+ vec![FilePart::new(0..length, 0, BlobRef::new(path, content))?],
+ )
+ }
+ .await;
+ if result.is_err() {
+ let _ = writer.abort().await;
+ }
+ result
+ }
+
+ /// Copy and verify a complete file with bounded transfer buffers.
+ ///
+ /// Every referenced blob is read in full, including bytes outside a part,
+ /// because its digest covers the complete blob. The assembled logical
+ /// file is checked separately. Bytes delivered before this method returns
+ /// are provisional: on any error the caller must discard the destination.
+ /// This method does not flush, close, or sync the caller's destination.
+ pub async fn read_file(
+ &self,
+ file: &File,
+ destination: &mut (impl AsyncWrite + Unpin),
+ ) -> Result<()> {
+ let mut logical_hasher = blake3::Hasher::new();
+ for part in file.parts() {
+ let reference = part.blob();
+ let path = data_path(reference)?;
+ let reader = self
+ .operator
+ .reader_with(path)
+ .chunk(BUFFER_BYTES)
+ .await
+ .map_err(read_error)?;
+ let mut stream = reader.into_stream(..).await.map_err(read_error)?;
+ let mut hasher = blake3::Hasher::new();
+ let mut offset = 0_u64;
+ let start = part.blob_offset();
+ let end = start + (part.range().end - part.range().start);
+ while let Some(buffer) = stream.try_next().await.map_err(read_error)? {
+ for bytes in buffer {
+ let next = offset
+ .checked_add(bytes.len() as u64)
+ .filter(|next| *next <= reference.content().length())
+ .ok_or_else(|| {
+ Error::corrupt("read YinYang file", "blob exceeds its declared length")
+ })?;
+ hasher.update(&bytes);
+ let copy_start = start.max(offset);
+ let copy_end = end.min(next);
+ if copy_start < copy_end {
+ let selected =
+ &bytes[(copy_start - offset) as usize..(copy_end - offset) as usize];
+ logical_hasher.update(selected);
+ destination.write_all(selected).await.map_err(|error| {
+ Error::from_io("write YinYang file destination", error)
+ })?;
+ }
+ offset = next;
+ }
+ }
+ if ContentId::new(hasher.finalize().into(), offset) != reference.content() {
+ return Err(Error::corrupt(
+ "read YinYang file",
+ "blob does not match its reference",
+ ));
+ }
+ }
+ if logical_hasher.finalize().as_bytes() != file.content().digest() {
+ return Err(Error::corrupt(
+ "read YinYang file",
+ "logical content digest does not match",
+ ));
+ }
+ Ok(())
+ }
+
+ pub(crate) async fn verify_new_files(&self, previous: &Tree, next: &Tree) -> Result<()> {
+ let previous = previous
+ .iter()
+ .map(|(_, node)| (node.id(), node))
+ .collect::<BTreeMap<_, _>>();
+ for (_, node) in next.iter() {
+ if let NodeBody::File(file) = node.body() {
+ if previous
+ .get(&node.id())
+ .is_some_and(|old| old.body() == node.body())
+ {
+ continue;
+ }
+ self.read_file(file, &mut tokio::io::sink()).await?;
+ }
+ }
+ Ok(())
+ }
+}
+
+fn data_path(reference: &BlobRef) -> Result<&str> {
+ let path = std::str::from_utf8(reference.as_bytes())
+ .map_err(|_| Error::corrupt("read YinYang file", "invalid data reference"))?;
+ let valid = path.strip_prefix(DATA_PREFIX).is_some_and(|id| {
+ id.len() == 32
+ && id
+ .bytes()
+ .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte))
+ });
+ if !valid {
+ return Err(Error::corrupt(
+ "read YinYang file",
+ "invalid data reference",
+ ));
+ }
+ Ok(path)
+}
+
+fn read_error(error: opendal::Error) -> Error {
+ if error.kind() == opendal::ErrorKind::NotFound {
+ Error::corrupt("read YinYang file", "referenced data is missing")
+ } else {
+ Error::from_storage("read YinYang file", error)
+ }
+}
diff --git a/crates/core/src/error.rs b/crates/core/src/error.rs
index fb76738..5855a67 100644
--- a/crates/core/src/error.rs
+++ b/crates/core/src/error.rs
@@ -25,6 +25,7 @@ pub enum ErrorKind {
NotFound,
Unsupported,
Storage,
+ Io,
}
/// Error returned by the YinYang Format core.
@@ -71,6 +72,10 @@ impl Error {
pub(crate) fn from_storage(operation: &'static str, error: opendal::Error) -> Self {
Self::new(ErrorKind::Storage, operation, error.to_string())
}
+
+ pub(crate) fn from_io(operation: &'static str, error: std::io::Error) -> Self {
+ Self::new(ErrorKind::Io, operation, error.to_string())
+ }
}
impl fmt::Display for Error {
diff --git a/crates/core/src/lib.rs b/crates/core/src/lib.rs
index 028753b..55ad402 100644
--- a/crates/core/src/lib.rs
+++ b/crates/core/src/lib.rs
@@ -17,6 +17,7 @@
//! YinYang Format values, OpenDAL persistence, and publication state machine.
+mod content;
mod error;
mod filesystem;
mod identity;
diff --git a/crates/core/src/publication.rs b/crates/core/src/publication.rs
index 39799b3..f7ebb34 100644
--- a/crates/core/src/publication.rs
+++ b/crates/core/src/publication.rs
@@ -52,7 +52,7 @@ pub enum CommitOutcome {
#[derive(Debug)]
pub struct Fs {
root: NodeId,
- operator: opendal::Operator,
+ pub(crate) operator: opendal::Operator,
}
impl Fs {
@@ -137,6 +137,8 @@ impl Fs {
.version
.tree()
.validate_successor(version.tree(), self.root)?;
+ self.verify_new_files(observed.tree(), version.tree())
+ .await?;
let number = version.number();
let blob = write_version(&self.operator, &version).await?;
match replace_head(&self.operator, &observed.head, &blob).await {
diff --git a/crates/core/tests/core.rs b/crates/core/tests/core.rs
index ae5926e..0ec0d4e 100644
--- a/crates/core/tests/core.rs
+++ b/crates/core/tests/core.rs
@@ -38,6 +38,8 @@ struct TestState {
fail_after_head_write: bool,
stat_calls: u64,
version_write_calls: u64,
+ fail_data_close: bool,
+ data_aborts: u64,
}
#[derive(Clone, Debug)]
@@ -274,6 +276,12 @@ impl oio::Write for TestWriter {
async fn close(&mut self) -> opendal::Result<Metadata> {
let mut state = self.state.lock().unwrap();
+ if self.path.starts_with(".yinyang/data/") && state.fail_data_close {
+ return Err(opendal::Error::new(
+ opendal::ErrorKind::Unexpected,
+ "data close failed",
+ ));
+ }
let current = state.objects.get(&self.path);
if self.if_not_exists && current.is_some() {
return Err(opendal::Error::new(
@@ -310,6 +318,9 @@ impl oio::Write for TestWriter {
}
async fn abort(&mut self) -> opendal::Result<()> {
+ if self.path.starts_with(".yinyang/data/") {
+ self.state.lock().unwrap().data_aborts += 1;
+ }
self.bytes.clear();
Ok(())
}
@@ -362,6 +373,300 @@ fn advance_root_membership(tree: &mut Tree) {
);
}
+fn add_file(tree: &Tree, name: &str, file: File) -> Tree {
+ let mut tree = tree.clone();
+ advance_root_membership(&mut tree);
+ tree.insert(
+ Path::new(name).unwrap(),
+ Node::file(NodeId::generate(), Generation::FIRST, false, file),
+ );
+ tree
+}
+
+#[tokio::test]
+async fn persists_file_bytes_across_reopen_and_old_versions() {
+ let backend = TestBackend::default();
+ let fs = Fs::create(backend.operator()).await.unwrap();
+ let bytes = (0..900_000).map(|i| (i % 251) as u8).collect::<Vec<_>>();
+ let file = fs.write_file(&mut bytes.as_slice()).await.unwrap();
+ let initial = fs.observe().await.unwrap();
+ let id = CommitId::generate();
+ let tree = add_file(initial.tree(), "large", file.clone());
+ backend.fail_next_head_write_after_success();
+ assert_eq!(
+ fs.commit(&initial, id, tree.clone()).await.unwrap(),
+ CommitOutcome::Committed { version: 1 }
+ );
+ assert_eq!(
+ fs.commit(&initial, id, tree).await.unwrap(),
+ CommitOutcome::Committed { version: 1 }
+ );
+ let reopened = Fs::open(backend.operator()).await.unwrap();
+ let observed = reopened.observe().await.unwrap();
+ let NodeBody::File(persisted) = observed
+ .tree()
+ .get(&Path::new("large").unwrap())
+ .unwrap()
+ .body()
+ else {
+ panic!()
+ };
+ let mut output = Vec::new();
+ reopened.read_file(persisted, &mut output).await.unwrap();
+ assert_eq!(output, bytes);
+ let replacement = fs.write_file(&mut &b"new"[..]).await.unwrap();
+ let old_node = observed.tree().get(&Path::new("large").unwrap()).unwrap();
+ let mut next = observed.tree().clone();
+ next.insert(
+ Path::new("large").unwrap(),
+ Node::file(
+ old_node.id(),
+ old_node.generation().next().unwrap(),
+ false,
+ replacement,
+ ),
+ );
+ fs.commit(&observed, CommitId::generate(), next)
+ .await
+ .unwrap();
+ output.clear();
+ reopened.read_file(&file, &mut output).await.unwrap();
+ assert_eq!(output, bytes);
+}
+
+#[tokio::test]
+async fn reads_parts_offsets_and_checks_the_logical_digest() {
+ let fs = Fs::create(TestBackend::default().operator()).await.unwrap();
+ let source = fs.write_file(&mut &b"0123456789"[..]).await.unwrap();
+ let blob = source.parts()[0].blob().clone();
+ let parts = vec![
+ FilePart::new(0..3, 6, blob.clone()).unwrap(),
+ FilePart::new(3..5, 1, blob).unwrap(),
+ ];
+ let file = File::new(
+ ContentId::new(blake3::hash(b"67812").into(), 5),
+ parts.clone(),
+ )
+ .unwrap();
+ let mut output = Vec::new();
+ fs.read_file(&file, &mut output).await.unwrap();
+ assert_eq!(output, b"67812");
+ let wrong = File::new(ContentId::new([0; 32], 5), parts).unwrap();
+ assert_eq!(
+ fs.read_file(&wrong, &mut tokio::io::sink())
+ .await
+ .unwrap_err()
+ .kind(),
+ ErrorKind::Corrupt
+ );
+}
+
+#[tokio::test]
+async fn empty_files_need_no_data_objects() {
+ let backend = TestBackend::default();
+ let fs = Fs::create(backend.operator()).await.unwrap();
+ let empty = fs.write_file(&mut &b""[..]).await.unwrap();
+ assert!(empty.parts().is_empty());
+ fs.read_file(&empty, &mut tokio::io::sink()).await.unwrap();
+ assert!(
+ !backend
+ .state
+ .lock()
+ .unwrap()
+ .objects
+ .keys()
+ .any(|path| path.starts_with(".yinyang/data/"))
+ );
+}
+
+#[tokio::test]
+async fn missing_or_corrupt_data_cannot_be_published() {
+ for corruption in ["missing", "digest", "short", "long"] {
+ let backend = TestBackend::default();
+ let fs = Fs::create(backend.operator()).await.unwrap();
+ let file = fs.write_file(&mut &b"content"[..]).await.unwrap();
+ let path = std::str::from_utf8(file.parts()[0].blob().as_bytes()).unwrap();
+ {
+ let mut state = backend.state.lock().unwrap();
+ if corruption == "missing" {
+ state.objects.remove(path);
+ } else {
+ let bytes = &mut state.objects.get_mut(path).unwrap().bytes;
+ match corruption {
+ "digest" => bytes[0] ^= 1,
+ "short" => {
+ bytes.pop();
+ }
+ _ => bytes.push(0),
+ }
+ }
+ }
+ let observed = fs.observe().await.unwrap();
+ let error = fs
+ .commit(
+ &observed,
+ CommitId::generate(),
+ add_file(observed.tree(), "bad", file),
+ )
+ .await
+ .unwrap_err();
+ assert_eq!(error.kind(), ErrorKind::Corrupt, "{corruption}: {error}");
+ assert_eq!(fs.observe().await.unwrap(), observed);
+ }
+}
+
+#[tokio::test]
+async fn failed_upload_aborts_and_never_changes_head() {
+ let backend = TestBackend::default();
+ let fs = Fs::create(backend.operator()).await.unwrap();
+ let observed = fs.observe().await.unwrap();
+ backend.state.lock().unwrap().fail_data_close = true;
+ assert_eq!(
+ fs.write_file(&mut &b"content"[..])
+ .await
+ .unwrap_err()
+ .kind(),
+ ErrorKind::Storage
+ );
+ assert_eq!(backend.state.lock().unwrap().data_aborts, 1);
+ assert_eq!(fs.observe().await.unwrap(), observed);
+}
+
+#[tokio::test]
+async fn invalid_data_locations_are_rejected() {
+ let fs = Fs::create(TestBackend::default().operator()).await.unwrap();
+ for path in [".yinyang/head", ".yinyang/data/../head", "external"] {
+ let content = ContentId::new(blake3::hash(b"x").into(), 1);
+ let file = File::new(
+ content,
+ vec![FilePart::new(0..1, 0, BlobRef::new(path, content)).unwrap()],
+ )
+ .unwrap();
+ assert_eq!(
+ fs.read_file(&file, &mut tokio::io::sink())
+ .await
+ .unwrap_err()
+ .kind(),
+ ErrorKind::Corrupt
+ );
+ }
+}
+
+#[tokio::test]
+async fn verifies_unselected_blob_bytes_and_reports_destination_errors() {
+ let backend = TestBackend::default();
+ let fs = Fs::create(backend.operator()).await.unwrap();
+ let uploaded = fs.write_file(&mut &b"abcdef"[..]).await.unwrap();
+ let blob = uploaded.parts()[0].blob().clone();
+ let file = File::new(
+ ContentId::new(blake3::hash(b"bc").into(), 2),
+ vec![FilePart::new(0..2, 1, blob.clone()).unwrap()],
+ )
+ .unwrap();
+ let (mut sink, peer) = tokio::io::duplex(1);
+ drop(peer);
+ assert_eq!(
+ fs.read_file(&file, &mut sink).await.unwrap_err().kind(),
+ ErrorKind::Io
+ );
+ backend
+ .state
+ .lock()
+ .unwrap()
+ .objects
+ .get_mut(std::str::from_utf8(blob.as_bytes()).unwrap())
+ .unwrap()
+ .bytes[5] ^= 1;
+ assert_eq!(
+ fs.read_file(&file, &mut tokio::io::sink())
+ .await
+ .unwrap_err()
+ .kind(),
+ ErrorKind::Corrupt
+ );
+}
+
+#[tokio::test]
+async fn source_failure_after_upload_starts_aborts_the_writer() {
+ struct FailingSource(bool);
+ impl tokio::io::AsyncRead for FailingSource {
+ fn poll_read(
+ mut self: std::pin::Pin<&mut Self>,
+ _: &mut std::task::Context<'_>,
+ buffer: &mut tokio::io::ReadBuf<'_>,
+ ) -> std::task::Poll<std::io::Result<()>> {
+ if self.0 {
+ return std::task::Poll::Ready(Err(std::io::Error::other("source failed")));
+ }
+ self.0 = true;
+ buffer.put_slice(b"started");
+ std::task::Poll::Ready(Ok(()))
+ }
+ }
+ let backend = TestBackend::default();
+ let fs = Fs::create(backend.operator()).await.unwrap();
+ assert_eq!(
+ fs.write_file(&mut FailingSource(false))
+ .await
+ .unwrap_err()
+ .kind(),
+ ErrorKind::Io
+ );
+ assert_eq!(backend.state.lock().unwrap().data_aborts, 1);
+ assert_eq!(fs.observe().await.unwrap().version().number(), 0);
+}
+
+#[tokio::test]
+#[ignore = "requires an isolated S3 bucket configured with YINYANG_S3_* variables"]
+async fn s3_file_publication_and_reopen() {
+ opendal::install_default();
+ let config = std::env::vars()
+ .filter_map(|(key, value)| {
+ key.strip_prefix("YINYANG_S3_")
+ .map(|key| (key.to_ascii_lowercase(), value))
+ })
+ .collect::<Vec<_>>();
+ let operator = Operator::via_iter("s3", config).unwrap();
+ let fs = Fs::create(operator.clone()).await.unwrap();
+ let bytes = vec![37; 12 * 1024 * 1024 + 17];
+ let file = fs.write_file(&mut bytes.as_slice()).await.unwrap();
+ let observed = fs.observe().await.unwrap();
+ let name = format!("file-{}", uuid::Uuid::new_v4().simple());
+ let tree = add_file(observed.tree(), &name, file);
+ let id = CommitId::generate();
+ assert!(matches!(
+ fs.commit(&observed, id, tree.clone()).await.unwrap(),
+ CommitOutcome::Committed { .. }
+ ));
+ assert!(matches!(
+ fs.commit(&observed, id, tree).await.unwrap(),
+ CommitOutcome::Committed { .. }
+ ));
+ assert!(matches!(
+ fs.commit(
+ &observed,
+ CommitId::generate(),
+ add_directory(observed.tree(), "loser", NodeId::generate())
+ )
+ .await
+ .unwrap(),
+ CommitOutcome::Conflict { .. }
+ ));
+ let reopened = Fs::open(operator).await.unwrap();
+ let current = reopened.observe().await.unwrap();
+ let NodeBody::File(file) = current
+ .tree()
+ .get(&Path::new(name).unwrap())
+ .unwrap()
+ .body()
+ else {
+ panic!()
+ };
+ let mut output = Vec::new();
+ reopened.read_file(file, &mut output).await.unwrap();
+ assert_eq!(output, bytes);
+}
+
#[tokio::test]
async fn creates_and_reopens_one_filesystem() {
let backend = TestBackend::default();
@@ -647,7 +952,7 @@ async fn requires_node_generation_for_file_changes() {
node,
Generation::FIRST,
false,
- File::new(ContentId::new([1; 32], 0), Vec::new()).unwrap(),
+ filesystem.write_file(&mut &b"original"[..]).await.unwrap(),
),
);
filesystem
diff --git a/specs/yinyang-format.md b/specs/yinyang-format.md
index 9ac4a92..2aafc25 100644
--- a/specs/yinyang-format.md
+++ b/specs/yinyang-format.md
@@ -71,6 +71,7 @@ YinYang owns these paths below the operator root:
```text
.yinyang/head
.yinyang/versions/<object-id>
+.yinyang/data/<object-id>
```
`<object-id>` is a generated UUID encoded as 32 lowercase hexadecimal
@@ -85,6 +86,34 @@ condition mismatch is a publication conflict. Missing, truncated,
length-mismatched, or unverifiable referenced objects are corrupt data. Direct
external writes under `.yinyang/` are outside the format contract.
+## File content IO
+
+`Fs::write_file` consumes an asynchronous byte source and returns a `File`
+only after the backend acknowledges the immutable data object. It writes raw
+bytes at a fresh UUID key while computing BLAKE3 and length with bounded
+buffers. A non-empty file has one part spanning that object; an empty file
+has the BLAKE3 digest of empty bytes and no parts or object. A failed upload
+returns no file and attempts to abort the writer. Cancellation or an ambiguous
+close failure may leave unreachable data, never a published namespace entry.
+
+`Fs::read_file` streams into a caller-owned destination. Each part reads and
+verifies its entire source blob, including bytes outside the selected range;
+the assembled logical file digest is also verified. Repeated references may
+read the same blob again. Memory use does not grow with file size. Missing
+data, invalid data keys, length mismatches, and digest mismatches are `Corrupt`;
+other backend failures are `Storage`, and source/destination IO failures are
+`Io`. Bytes delivered before successful completion are provisional. Callers
+must discard them on failure and own destination flushing and durability.
+
+Before publishing, `Fs::commit` reads and verifies each new or changed file
+representation. Unchanged file bodies under the same node identity need no
+new read. This includes files constructed by callers rather than uploaded
+through `write_file`, and prevents publication of incomplete references. The
+initial implementation trades an extra read of new data for this guarantee.
+The guarantee assumes immutable data is not externally modified or deleted.
+Head conflicts can leave complete but unreachable uploads. There is no garbage
+collection yet; callers must not delete objects referenced by older versions.
+
## Persistent encoding
The payload uses [Borsh encoding](https://borsh.io/). Integers use little-endian